* Jose V Beneyto (sepen@crux-arm.nu) wrote:
Hi Victor,
* Victor Martinez (pitillo@crux-arm.nu) wrote:
Hello, I have been doing some test with native compilations under handhelds (at the moment I'm trying on htc-prophet). nice, I tried on my jornada too but nothing serious, just for fun ;D I noticed what Fredick told me about the -pipe flag, which seems to kill memory (only with 64Mb). I want to know your opinions about this and another topic related to native compilations and the commits in native git repos. I think it can be a good idea to remove the -pipe flags in ports which have it hardcoded (like perl). In this way we can let native building work with devices which don't have too much amount of ram. well since pda's are not designed for native compilations IMHO a comment in a doc would be enough. Anyways we should try to keep ports as transparent as possible and we should let the user the freedom to use his own flags, so I like more the idea of replace flags instead of remove some flags (all
On 05/17/10 09:07, pitillo@crux-arm.nu wrote: ports must accept user defined flags in pkgmk.conf whenever be possible)
well, pda's aren't the aim, but they are another option to do that (nfs exports in newer devices which their CPU are around 1Ghz and they come with more than 512MB of RAM, I don't see the point about don't make native builds there) About flags, I made a difference between user flags and hardcoded flags, like in perl, which has hardcoded -pipe and it adds that to the user's flags (Btw, perl doesn't build fine without the -pipe hardcoded flag). These kind of cases will be the less.
We can see this errors reported at the end of the pkgbuild log: "cc: Internal error: Killed (program cc1) Please submit a full bug report.
Related to ram are sources too, xz and lzma needs more ram to be uncompressed (their size is smaller that bz2 sources) and then it's impossible in devices with small amount of ram. A full report about xz/lzma problem: lzma: =======> Building '/nfs/crux/pkg/curl#7.20.1-1.pkg.tar.gz'. bsdtar -p -o -C /nfs/crux/work/src -xf /nfs/crux/src/curl-7.20.1.tar.lzma unlzma: (stdin): Memory usage limit reached unlzma: Limit was 22 MiB, but 33 MiB would have been needed bsdtar: Unrecognized archive format bsdtar: Child process exited with status 1 bsdtar: Error exit delayed from previous errors. =======> ERROR: Building '/nfs/crux/pkg/curl#7.20.1-1.pkg.tar.gz' failed. xz: bsdtar -p -o -C /nfs/crux/work/src -xf /nfs/crux/src/coreutils-8.5.tar.xz unxz: (stdin): Memory usage limit reached unxz: Limit was 22 MiB, but 65 MiB would have been needed bsdtar: Unrecognized archive format bsdtar: Child process exited with status 1 bsdtar: Error exit delayed from previous errors. =======> ERROR: Building '/nfs/crux/pkg/coreutils#8.5-1.pkg.tar.gz' failed.
Using gz/bz2 sources avoid this problem, but we can get the problem exposed above. same on my jornada 720, but only 11MiB:
[...] =======> Building '/usr/pkg/diffutils#3.0-1.pkg.tar.gz'. bsdtar -p -o -C /dev/shm/diffutils-work/src -xf /usr/pkg/src/diffutils-3.0.tar.xz unxz: (stdin): Memory usage limit reached unxz: Limit was 11 MiB, but 65 MiB would have been needed [...]
well I found something interesting in the code (see Changelog line) 11MB is around 30% of 32MB (which has the j720 device) and 22MB is around 30% of 64MB (your htc-prophet device)
[...] $ cd /dev/shm/xz-work/src/xz-4.999.9beta $ grep 'Memory usage limit' -r * ChangeLog: - Memory usage limit is now about 30 % for uncompression po/xz.pot:msgid "Memory usage limit (%<PRIu64> MiB) is too small for the given filter setup (%<PRIu64> MiB)" po/xz.pot:msgid "Memory usage limit reached" src/liblzma/common/alone_decoder.c: /// Memory usage limit src/liblzma/common/index_decoder.c: /// Memory usage limit src/liblzma/common/stream_decoder.c: /// Memory usage limit src/liblzma/api/lzma/base.h: * \brief Memory usage limit was reached src/liblzma/api/lzma/container.h: * - LZMA_MEMLIMIT_ERROR: Memory usage limit was reached. src/liblzma/api/lzma/index.h: * - LZMA_MEMLIMIT_ERROR: Memory usage limit was reached. src/liblzma/api/lzma/lzma.h: * - Memory usage limit src/xz/coder.c: message_fatal(_("Memory usage limit (%" PRIu64 " MiB) is too small " src/xz/hardware.c:/// Memory usage limit src/xz/message.c: return _("Memory usage limit reached"); src/xzdec/xzdec.c: msg = "Memory usage limit reached"; [...]
and this is the changeset which introduced it:
[...] commit 2672bcc9f85ba28ff648e092e9eb4cd9e69ce418 Author: Lasse Collin <lasse.collin@tukaani.org> Date: Sun Mar 7 13:29:28 2010 +0200
Increase the default memory usage limit on "low-memory" systems.
Previously the default limit was always 40 % of RAM. The new limit is a little bit more complex:
- If 40 % of RAM is at least 80 MiB, 40 % of RAM is used as the limit.
- If 80 % of RAM is over 80 MiB, 80 MiB is used as the limit.
- Otherwise 80 % of RAM is used as the limit.
This should make it possible to decompress files created with "xz -9" on more systems. Swapping is generally more expected on systems with less RAM, so higher default limit on them shouldn't cause too bad surprises in terms of heavy swapping. Instead, the higher default limit should reduce the number of bad surprises when it used to prevent decompression of files created with "xz -9". The DoS prevention system shouldn't be a DoS itself.
I hope this can be interesting. Waiting your comments and suggestions. maybe we could try to find a solution/patch (maybe I'll play at night), talk with xz developers about that, or as a last resort try an alternative
Note that even with the new default limit, a system with 64 MiB RAM cannot decompress files created with "xz -9" without user overriding the limit. This should be OK, because if xz is going to need more memory than the system has RAM, it will run very very slowly and thus it's good that user has to override the limit in that case. [...] like you said (tar.bz2 instead of xz sources)
good research Jose. I think it would be better to change sources to keep them more compat with old devices (or with less resources, RAM in this case). Seems that the aim of xz/lzma is reduce the space (compressing more) but using more ram at runtime. I really prefer to change sources instead of patching. Waiting comments or suggestions.
Regards, Victor.
Best regards,
-- Jose V Beneyto | http://sepen.mine.nu/
--- Learning bit by bit Victor Martinez | http://lokalix.dyndns.org