[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: finally subscribed

On Wed, Aug 30, 2000 at 02:02:12AM +0200, Hubert Feyrer wrote:

> > My effort so far has been
> > directed:
> > - to speed up make (and bsd.port.mk), with very good results. The
> > infrastructure  itself is about three to four times as fast as it was some
> > time ago.  Besides plugging in a very fast hashing library (with features
> > that make it better for make than what's currently in other BSDs), there also
> > was a lot of streamlining in the list library (which removes most of the
> > dynamic aspects of lists, which are thoroughly unneeded in make), and
> > a rewrite of the parser (in progress). This means that lots of variables,
> > .for loops, or loads of comments don't have the impact on Makefiles they
> > used to have.
> > - to improve POSIX compatibility. There was a suffixes bug that was fixed
> > six months ago, and a very important change that allows for variables
> > set on the command line to be passed to submakes (which simplifies faking
> > tremendously).
> > - to stamp out various bugs and add small features, such as regularizing
> > variable lookup in the context of conditional, or the recently proposed
> > `looking up targets along the VPATH' for sanity reasons (for which I have
> > to write a few lines of code yet).
> > 
> > This is a somewhat major endeavor. The complete diff for that was larger
> > than make source itself at some point (yep, I pretty much read all of
> > make's source and rewrote a large portion of it). It is now nearing the
> > point where I don't see obvious improvements.
> Er, I read that as a lot of internal code change (new hash lib, ...), and
> one change visible to the ports system, the passing down of variables that
> you're using for the fake builds? (not to belittle your work, I'd never
> dare to touch make sources :)

The speed issues have a large impact on bsd.port.mk: more intricate stuff
becomes easy to do. One issue I completely forgot about is decent 
error messages. OpenBSD's flavor of make now displays the directory in which
it fails when it does, along with the line number and the Makefile name,
which makes it much easier to locate problems, even for rules prefixed
with @.

> What other changes did you make that made OpenBSD make(1) different from
> what it was before (and what one might use FreeBSD or NetBSD make(1) for)?

I'll have to check more closely. The suffixes fix allowed to use pmake
for one more port that used to depend on gnu-make. There were some conditional
bug-fixes which mean some work-around in bsd.port.mk are no longer necessary.
I'll try to find out which bug, specifically, but if you get OpenBSD
bsd.port.mk thru NetBSD's make, you'll notice it fails in several places.

One recent change is the lookup of variables in the expansion of for loops
(revision 1.20 of OpenBSD for.c):
this used to start in the global context, instead of including VAR_CMD as
per other variables.  

Consider the Makefile fragment:
.for B in $A

invoked with make A=5
then B will take the value 3 in that for loop, even when A will be uniformly
5 throughout the Makefile.

This had downright confusing consequences throughout the ports tree, where
loops are rather frequent.

> perl does the compilation each time you call the script, not once, so I
> doubt that'll be a win. Esp. on old & slow machines. (think vax, 25MHz
> m68k or 16MHz i386/sx).

Perl's compilation is a very simple affair (syntax analysis mostly). This
is non-intuitive, but there are perl scripts that perform faster than their
C counterparts, and, as an Amiga owner, slow machines are still a concern
for me.

> > or compartments for safety (see Safe(3p) ), 

> # man -s3p Safe
> No entry for Safe in section(s) 3p of the manual.
> It's fine to use such libs if one has them in tree, using them for a
> project that most targets don't have it won't help us. 

Safe is a part of perl5.  Either you don't have perl installed, or maybe
the documentation is only in pod format ? in which case perldoc Safe ought
to work.

> First, I don't think that any of the pkg_* tools are time critical.
> Then, the C code makes shell-callouts for things that aren't trivial to
> code - I don't know if you intend to rewrite tar(1) in perl ... :-)

Rewriting only the part of tar(1) that packages need is not hard, considering
that all packages are built with the same options of tar, and hence you don't
need to interpret all archives. Also, consider that pkg_* only need to deal
with complete archives, and don't have to try to recover information from 
corrupt archives (since you can always use the real tar for that).

Considering the difficulty of communicating with tar, with its unwieldy set
of options, and the limitations of -C and otherwise, a minial tar in perl
is smaller than the corresponding interface-to-tar code in pkg_*...

Yes, I intend to rewrite the extraction part of tar in perl. In fact, 
I already have it. And pkg_add would already be converted if it were not
for some side issue that's taking most of my free time (binutils).

> Anyways, assuming'd have the current pkg_* rewritten in perl, what would
> you do next?

Use the extra flexibility and ease of prototyping to see how far the ball
can go. I don't have too much written down yet, as I'm still trying out
ideas, and things haven't frozen yet.
	Marc Espie		
|anime, sf, juggling, unicycle, acrobatics, comics...
|AmigaOS, OpenBSD, C++, perl, Icon, PostScript...
| `real programmers don't die, they just get out of beta'

To unsubscribe: send mail to <majordomo@unixathome.org>
with "unsubscribe bsdports" in the body of the message