channel logs for 2004 - 2010 are archived at http://tunes.org/~nef/logs/old/ ·· can't be searched
#osdev2 = #osdev @ Libera from 23may2021 to present
#osdev @ OPN/FreeNode from 3apr2001 to 23may2021
all other channels are on OPN/FreeNode from 2004 to present
http://bespin.org/~qz/search/?view=1&c=osdev&y=19&m=2&d=7
12:02:59 <jmp9> Okay i got pointer 0x00010000 in EBX
12:03:04 <jmp9> so it's looks like normal
12:03:15 <jmp9> problem was in my xprint function which prints pointers
12:03:19 <jmp9> :p
12:15:27 <klange> jmp9: The memory map entry's size component is a bit of an oddity.
12:19:33 <jmp9> Okay
12:19:34 <jmp9> https://hastebin.com/giwebevino.cpp
12:19:37 <jmp9> this is my code
12:19:39 <jmp9> and it doesn't work
12:19:50 <jmp9> I followed GRUB multiboot specification
12:21:05 <klange> Lemme pull up mine, I have an implementation of both sides.
12:21:28 <klange> https://github.com/klange/toaruos/blob/master/kernel/main.c#L135
12:21:36 <haxmoLLC> sighhhh their "link to text" button doesn't work without JS
12:23:30 <jmp9> https://imgur.com/a/LOKFhSC
12:23:37 <haxmoLLC> *gasps* ONE SPACE!?
12:24:51 <haxmoLLC> klange you madman, oh hey i saw this OS on reddit a couple months ago, that's you huh? nice f'n work!
12:25:00 <klange> ?
12:25:04 <klange> Those are tabs.
12:25:16 <haxmoLLC> oh github screwed up then
12:25:28 <klange> github did not screw up, and they printed as tabs in the HTML.
12:25:39 <jmp9> Okay okay okay okay
12:25:59 <jmp9> 28 - 40 | syms | (present if flags[4] or | | flags[5] is set)
12:26:02 <haxmoLLC> indeed the "raw" link is actualy legible :)
12:26:03 <jmp9> this in specification
12:26:09 <jmp9> syms
12:27:50 <haxmoLLC> they're probably doing that trendy "reformat everything after the page has already loaded" thing
12:29:07 <nyc> Oh nice, or1k is actually available if I build qemu myself. And alpha.
12:30:57 <kingoffrance> if you screw around with alpha, lemme know if it is faster than es40 (also works on windows).
12:31:25 <nyc> kingoffrance: I'm very pro-RISC.
12:33:04 <klange> haxmoLLC: They're not and that page looks exactly the same with or without Javascript. What browser are you on? GitHub workings amazingly well without Javascript, even several of their dropdown menus work (because they use CSS tricks instead of JS)
12:33:09 <jmp9> uhm
12:33:11 <jmp9> I got this
12:33:27 <jmp9> 0x00000000 - 0x0009FC00
12:33:31 <jmp9> this is 0.6 megs
12:33:37 <jmp9> but kernel loads below 1 meg
12:33:38 <jmp9> wtf?
12:33:41 <klange> Well... the line highlight doesn't work. That's actually a shame, you can detect anchors with CSS.
12:33:48 <kingoffrance> nyc: assuming you also screw around with es40, i am not trying to order you around
12:33:52 <nyc> Wait a minute, is riscv in the mainstream qemu tree current?
12:34:04 <nyc> kingoffrance: I don't really know about es40.
12:34:09 <klange> jmp9: print your entire memory map, because that entry looks correct to me
12:36:07 <jmp9> yes, it works. I just wrong iterating
12:36:15 <jmp9> mmap = (mboot_memmap_t *) ((uintptr_t)mmap + mmap->size + sizeof(uintptr_t));
12:36:18 <jmp9> wtf????
12:36:25 <jmp9> that's not in osdev example
12:36:32 <jmp9> or multiboot specification
12:38:12 <klange> Yes it is.
12:38:13 <jmp9> ok nice, i got 64 megs as i specified in qemu options
12:38:13 <klange> mmap = (multiboot_memory_map_t *) ((unsigned long) mmap
12:38:14 <klange> + mmap->size + sizeof (mmap->size)))
12:38:25 <jmp9> AHHHHHHHHH
12:38:29 <jmp9> sizeof(mmap->size)
12:38:32 <jmp9> AWHDAWDAWDAWDAWD
12:38:55 <haxmoLLC> klange netsurf, your site renders nicely at least, until i click to go to gitlab which is a mess
12:39:09 <klange> haxmoLLC: netsurf is likely not rendering the tabs correctly
12:39:13 <haxmoLLC> NIH lib sounds very interesting
12:39:29 <klange> NIH project was completed, base ToaruOS is now 100% original code :)
12:40:01 <klange> And we can run Python and GCC and Quake again, so huge success.
12:40:11 <kingoffrance> nyc: i can vouch es40 is complete enough to run freebsd (5+ 6 i believe is when they dropped alpha) and tru64 and should do vms too. i havent tried qemu alpha. neither will run nt4 to my knowledge
12:40:38 <klange> netsurf's CSS and HTML implementations are both very outdated and incomplete, so it's understandable that they trip on modern pages even without javascript.
12:40:58 <haxmoLLC> it's interesting how that affects tabs becoming spaces
12:41:26 <haxmoLLC> sadly there are no other usable browsers
12:41:42 <haxmoLLC> maybe `links -G` but it's really gray and dull looking
12:42:40 <klange> What is the source of your inability to run a modern browser? Privacy concerns? Lack of an X11 or Wayland? I think there's a Webkit browser that will work against a Linux framebuffer.
12:43:06 <haxmoLLC> trust concerns
12:43:21 <klange> Uh, run a Gecko browser that isn't Firefox?
12:43:44 <klange> Why do you trust Netsurf any more than anything else?
12:43:57 <haxmoLLC> their code is readable and compiles in less thana minute
12:44:04 <nyc> kingoffrance: I guess I should target es40, then, but I'm having enough trouble getting to hello world on the bare metal.
12:44:15 <klange> Their code is ancient and broken and you can't even use most of the Internet.
12:44:25 <haxmoLLC> future improvements are slow, but i'm hopeful someday the outstanding issues will be addressed
12:45:00 <haxmoLLC> another thing is why do we bother standardizing any of this html js css crap
12:45:08 <haxmoLLC> if it's just up to google to do what they want
12:45:22 <haxmoLLC> they can hold all brosers hostage by breaking API's and adapting new ones on demand
12:45:37 <haxmoLLC> creating an obscenely large barrier of entry into the browser market
12:47:17 <nyc> kingoffrance: I also have plenty of targets just from qemu.
12:48:04 <nyc> kingoffrance: My basic plot is to put all the RISC arches on equal footing, and all of them ahead of x86.
12:52:50 <nyc`> kingoffrance: Hmm. VAX and 68k might be good too.
12:54:27 <nyc> kingoffrance: Call it an x86 unter alles approach. :)
01:20:56 <nyc> qemu IA64 seems to be 8 years dead.
01:22:47 <jmp9> 30 gb of porn
01:22:51 <jmp9> oh
01:22:53 <jmp9> wrong window :D
01:24:00 <nyc> jmp9: You may need to develop a more scalable filesystem for that.
01:24:39 <jmp9> huh
01:24:44 <jmp9> i'm VFAT on external hard drive
01:24:48 <jmp9> for 1 TB of my *stuff*
01:25:00 <jmp9> i'm using VFAT*
01:25:32 <jmp9> just split a hundreds thousands of pictures by 4 thousand pictures for each folder
01:25:37 <jmp9> to avoid file limit in current folder
01:32:44 <nyc> That's probably stressing the scalability limits of the directory data structures in VFAT.
01:44:15 <nyc> jmp9: The answer is that you need to develop a more scalable filesystem than even zfs.
01:45:15 <nyc> jmp9: Call it pornfs, capable of addressing 2**256 512B blocks and holding 2**256 files that are up to 2**256B in size etc.
01:51:56 <nyc> https://gts3.org/assets/papers/2016/min:fxmark.pdf (qemu is building...)
01:55:55 <kingoffrance> nyc: im not there yet either. my plan is basically 1) implement dynamic linker/loader/bootloader, run c on "bare metal" as much as possible 2) write assembler 3) use 2) as much as possible for anything i cant do from C
01:56:11 <kingoffrance> im very slow,but something funny that makes me glad i "waited"
01:56:29 <nyc> jmp9: You may even need to plan for a future of 2**1024 1MB blocks.
01:56:39 <kingoffrance> 2) would be "assemble from bare metal/"kernel' " basically i mean
01:57:17 <jmp9> fuck
01:57:20 <jmp9> my friend is asshole
01:57:45 <jmp9> sometime i'll fucking kill him
01:58:17 <jmp9> he told everyone my secret in our class chat
01:58:34 <jmp9> just because i make jokes of his ex
01:58:48 <klange> please keep your drama out of here ;)
01:58:59 <jmp9> its 3:58 am
01:59:33 <klange> ouch
02:00:24 <kingoffrance> https://www.freebsd.org/cgi/man.cgi?a.out(5) basically for me "Symbols map names to addresses (or more generally, strings to values)." my "name" will be a single-character in a "custom" charset, so i will not hardcode any particular charset in my "loader" stuff
02:00:31 <klys> alis doesn't have any popular drama channels. ##anime-drama has three users.
02:00:52 <kingoffrance> overkill perhaps, but i am sorta glad i am so slow/"abstract" in some ways
02:01:24 <kingoffrance> it comes down to how far can i get without writing own compiler/linker/etc. :/
02:02:01 <jmp9> haha anime
02:02:04 <kingoffrance> i might have to "post process' all my objects or something
02:02:50 <jmp9> also
02:03:06 <jmp9> it will be normal if I will drop to real mode every time I need to read sectors from drive?
02:03:30 <jmp9> i don't want to f*ck with USB because i want keep it simple
02:03:53 <jmp9> main purpose of my os will be loading some ELFs and working with FAT16
02:04:07 <klange> Real world drama has all moved to Discord.
02:04:16 <jmp9> haha i'm banned in discords
02:04:23 <jmp9> just because i sent photo of worms
02:04:35 <klange> What kind of worms?
02:04:44 <jmp9> "опарыши"
02:04:50 <nyc> I use Discord to practice speaking foreign languages.
02:04:57 <jmp9> maggots
02:05:11 <jmp9> they're so cute
02:05:20 <klange> Hm. Really depends on the context. A pile of maggots aren't all that disgusting, but if they're eating the wrong/right thing...
02:05:45 <jmp9> also "опыраши" aka maggots is russian local meme
02:05:55 <kingoffrance> (i have no intention to use a.out, i was just fleshing out "things needed for a process")
02:05:58 <jmp9> which popular in russian game hacking communities
02:10:16 <jmp9> ыщ
02:10:17 <jmp9> so
02:10:30 <jmp9> to page 4 gigs of virtual mem we need 4 MB for page tables and page directories
02:10:31 <jmp9> ?
02:11:03 <jmp9> sry i'm highh
02:11:53 <bcos_> For 4 GiB of virtual space (where only 1 KiB is actually "present" at any one time) you only need 4 KiB for one page table and 4 KiB for one page directory
02:12:59 <jmp9> page directories made for caching, right?
02:16:57 <klys> https://wiki.osdev.org/Paging
02:18:06 <nyc> I think the idea is that pagetables don't have to be fully populated.
02:20:27 <Ameisen> for regular binary data, what's generally these days the best compression algorithm when I don't care about compression speed, only decompression speed?
02:20:45 <Ameisen> snappy? brotli?
02:21:07 <Celelibi> I'd guess it depends on the particular structure of your binary data.
02:21:22 <Celelibi> You can't really compress anything with random data.
02:21:24 <nyc> Decompression doesn't usually take much in any of the algorithms.
02:23:08 <nyc> Speed of decompression really isn't an issue for any algorithm that I'm aware of.
02:32:56 <kingoffrance> for "newer" stuff i have noticed there are pbzip2 and perhaps pgzip variants (parallel)...so that would perhaps be one of my criteria, is there a multicore/threaded decompressor source available?
02:35:43 <nyc> kingoffrance: I think gzip breaks up the source into 64KB sections, so offsets are predictable and it's parallelizable.
02:36:59 <bcos_> (parallilizable, if and only if source is >= 64 KiB)
02:38:04 <bcos_> Breaking source into blocks effects compression ratio too (smaller blocks = worse compression)
02:46:03 <kingoffrance> heres a q. i have lots of vms, some disk images from real hardware. eg on real hardware i always zero drive first, then install os, then image, for smaller images
02:47:09 <kingoffrance> if i was going to restore an image to a real disk, is it faster to zero drive first, then only restore non-zeroed bytes, or just write entire image and let it write zeroes? at what size drive might it be worthwhile to "sparse restore"
02:47:37 <kingoffrance> or does this "depend on type of disk how fast it is to zero it"
02:48:23 <kingoffrance> ive thought about such a util, but not sure it is worth it
02:48:54 <kingoffrance> i imagine zeroing a drive is faster than writing data to it, but not sure it is worthwhile to do this
02:55:25 * bcos_ can't think of any actual hardware where zeroing is different to "writing data that is all zeros"
02:55:45 <bcos_> (and if you're writing data that's all zeros before writing that that's not zeros, then ...)
02:57:46 <Mutabah> bcos_: Flash cells?
03:00:49 <bcos_> No..
03:01:59 <bcos_> (although I was wondering if "trim" can be abused for tamper-proofing - what happens if you read a block that was trimmed/freed, and if you get the same data back if and only if the "freed" block isn't reused)
03:03:02 <bcos_> For completeness, hard disks often have a "secure erase" too, but I don't think they're required to leave the sector full of zeros (instead of any randomised pattern) and the performance (vs. writing zeros) would be horrible
03:04:48 <kingoffrance> there is scsi "format" command, i dont know what it does, but thinking along those lines, assuming something like that would zero things
03:06:19 <bcos_> Low level formats typically just write metadata (the "address and CRC" marks at start of sector) and skip the contents of each sector; and I think modern stuff doesn't support it (it's more common for floppy drives and ancient IDE)
03:07:03 <bcos_> (mostly because the pretend "CHS" doesn't really exist anymore either)
03:07:42 <bcos_> (meaning, number of sectors per track is different for different tracks, making it too hard to figure out how to format)
03:08:33 <bcos_> Dammit
03:09:14 <bcos_> Know my brain has decided to start thinking about "roll your own paging on disk" with sector/s reserved for the equivelent of page tables
03:09:23 <bcos_> *Now
03:10:16 <bcos_> (OS asks disk driver for "virtual sector #3", disk driver does some sneaky tricks to figure out which physical sector and then asks device for ..)
03:10:33 <klys> slow
03:10:40 <bcos_> Wait...
03:10:59 <bcos_> klys: Which VM is it? Some (most) support "sparse and/or growing disk images"
03:11:14 <bcos_> Ooops.. ^ @ kingoffrance
03:11:32 <kingoffrance> no, the only reason i really ever use real hardware is some old OSes i cant get rnning in qemu, or vice versa
03:11:39 <kingoffrance> so it is really only for "have to run on physical hardware" case
03:11:43 <jmp9> You know
03:11:49 <jmp9> Smoking kills your brain cells
03:11:56 <klys> not mine
03:12:01 <jmp9> Like overvoltage kills flash cells
03:13:11 <kingoffrance> basically, for this screwing around i avoid using qemu/virtualbox/etc. cow disk images, because doing this i can easily just dd to a real disk if necessary, from random linux livecd or whatever
03:14:47 <kingoffrance> maybe it just kils the weak cells, leaving a stronger, more efficient, streamlined brain system
03:14:52 <bcos_> jmp9: Over-voltage also causes brain cells to smoke - if you ever get your head stuck between high voltage power distribution lines you won't regret it (in a bad way)
03:15:17 <jmp9> Huh
03:15:33 <jmp9> That's why you shouldn't put your fingers to socket :-)
03:16:19 <bcos_> Well, no - you shouldn't put your fingers in a socket because you have to carve them down to make them small enough to fit
03:16:25 <klange> I pretty much exclusively deal in iso9660 and ustar these days.
03:16:40 <klange> And some FAT because EFI
03:17:14 <klys> https://cdn11.bigcommerce.com/s-915s2wha/images/stencil/500x659/products/932/1656/CFT1__74432.1444380326.jpg
03:19:00 <bcos_> klys: https://i.imgur.com/7gbIUhd.gif
03:21:21 <nyc> I need to figure out what's going on with filesystems. My big guess is that the top of the line things are xfs and zfs.
03:23:47 <klys> :o
03:26:40 <klange> No one actually knows anymore.
03:26:59 <klange> So don't worry, you're just as confused as the people actually working filesystems.
03:27:24 <haxmoLLC> ext2 is kind of small but meh gpl2
03:27:26 <klange> You were around when Reiser was tried for killing his wife, right?
03:27:41 <klange> haxmoLLC: ? Implement it yourself. The design and specification are not GPL2.
03:27:44 <nyc> klange: Mostly nobody heard about it while it was happening.
03:29:10 <haxmoLLC> i'm haxmo, not reimplementabsolutelyeverythingmo
03:29:21 <ybyourmom> I'm sure there are papers which have benchmark results for them
03:29:23 <klange> Then go grab someone else's non-GPL2 implementation.
03:29:39 <klange> There are dozens of BSD-licensed ext2s.
03:29:47 <haxmoLLC> great idea!
03:29:54 <klys> haxmollc: http://show.ing.me/0001gefi.zip
03:30:19 <haxmoLLC> that reminds me, i like how minix and netbsd are collaborating
03:30:40 <klange> idk, I think it means less variety and fewer independent complete projects.
03:30:48 <klange> Just another kernel to run BSD on.
03:31:15 <haxmoLLC> klys: what's this?
03:32:01 <klys> haxmollc, a pared down grub-efi tree that just loads and prints a file in ext4 from the efi shell
03:32:27 <haxmoLLC> ext4 is too large for me to consider
03:32:46 <haxmoLLC> currently
03:33:04 <klys> it's the grub ext2 driver, basically. i could have as well said ext2
03:33:41 <klange> Do you need write support?
03:33:52 <haxmoLLC> yeah of course
03:34:20 <klange> You can try to fix the bugs in mine. https://github.com/klange/toaruos/blob/master/modules/ext2.c
03:35:32 <haxmoLLC> i'll give it a look, bookmarked. that's a bit further down the road still
03:36:00 <klange> It's pretty easy to implement something on relative feature parity with ext2 on your own.
03:36:30 <klange> Which I've also been meaning to do...
03:36:41 <klys> yeah first you'd want a sector editor and then a big idea
03:38:05 <nyc> klys: A big idea?
03:38:13 <klys> like, inodes
03:38:32 <klange> block bitmap, inodes, and directory entries - that's all you need to get feature parity with the things that matter in ext2
03:38:38 <nyc> Just use B+ trees for everything.
03:38:51 <klys> btrfs does that?
03:39:04 <klange> The block group descriptors are useless, the backup superblocks are useless, and anything more advanced puts you beyond ext2.
03:39:07 <nyc> I've never seriously looked at btrfs.
03:50:55 <nyc> I think one needs to index directories as an ordered set of names that in turn map to file structures. One basically wants to be able to traverse in order as well as look up.
03:51:38 <Ameisen> I use btrfs on some of my systems since it makes it pretty darned easy to compress data (in the same fashion as ntfs compression) and also backup stuff.
03:51:42 <Ameisen> since it's all cow
03:51:58 <nyc> For files similar holds with file offsets to actual disk addresses.
03:53:11 <Ameisen> though the future, of course, is pifs
03:53:16 <nyc> pifs?
03:54:02 <nyc> Oh dear.
03:54:10 <nyc> The humor thing?
04:08:54 <Ameisen> humor?
04:08:57 <Ameisen> You doubt the abilities of pi?
04:09:12 <klange> All your data is in there somewhere.
04:09:28 <Ameisen> which is why Oracle is now suing for ownership of pi
04:09:31 <Ameisen> copyright infringement.
04:10:38 <nyc> I mostly just need to get beyond hello world on the various arches and up to loading userspace executables.
04:12:09 <klys> 3.14159265358979323846264338327950
04:12:19 <klys> thought you might like a free copy
04:12:36 <nyc> klys: That's a lot of porn.
04:17:02 <nyc> gdb'ing qemu didn't tell me as much as I hoped it would.
04:18:27 <mrvn> nyc: b+ trees need a lot of writes for every change
04:19:05 <nyc> mrvn: Not really. Most insertions and deletions don't require any rebalancing.
04:19:43 <mrvn> nyc: I assume you are doing a COW fs or you get horrible data loss on crash. And then any write means log(n) block updates.
04:20:37 <mrvn> You can journal instead but that has it's own set of problems.
04:20:59 <nyc> mrvn: That's technically shallower than radix trees, which don't have the opportunity to become shallower with greater contiguity.
04:21:36 <mrvn> it's theoretically and practicall much much deeper than arrays like ext2 has.
04:22:01 <nyc> Arrays for what?
04:22:08 <mrvn> nyc: everything.
04:22:08 <nyc> Directories?
04:22:24 <nyc> ext2 uses radix trees for file offset to block translation.
04:22:32 <mrvn> nyc: nope.
04:23:56 <nyc> What I'm reading says that it does.
04:24:02 <mrvn> small files have a simple array of blocks. Larger files have an array of arrays of blocks and even larger an array of arrays of arrays of blocks.
04:24:11 <mrvn> nyc: are you readin ext4?
04:24:35 <nyc> No.
04:24:55 <nyc> That's a radix tree.
04:25:26 <nyc> I suppose it's using a trick to make it shallower for small files.
04:25:51 <mrvn> nyc: it's more like a fibnaccy tree
04:26:18 <mrvn> nyc: the first blocks are shallow and the more blocks you get the deeper a tree you have.
04:26:26 <nyc> Fibonacci tree? AVL trees are the ones with Fibonacci affairs.
04:27:10 <nyc> That doesn't actually make a difference relative to B+ trees.
04:27:26 <mrvn> a radix tree would also support variable key lengths and balance themself. The ext2 "tree" is not balanced or optimized at all.
04:27:58 <nyc> B+ trees are shallow for small files too.
04:28:33 <nyc> mrvn: Radix trees don't do all that. That's more like a hash trie.
04:28:57 <mrvn> except if you look at btrfs then you see that file size is totally irelevant. There is only one B+ tree for all files together (iirc)
04:29:06 <mrvn> nyc: https://en.wikipedia.org/wiki/Radix_tree
04:29:18 <cryptonix2> *Fibonacci
04:29:24 <mrvn> nyc: nice big fat example image on the right.
04:33:48 <nyc> btrfs' one B+ tree for everything is nuts.
04:34:23 <nyc> mrvn: It's a pretty weird description.
04:34:23 <mrvn> better than one per file
04:34:36 <nyc> One per file would actually make sense.
04:34:54 <mrvn> not if you consider updates
04:36:23 <mrvn> having a b+ tree of b+ trees would just add more complexity and increase the number of writes needed for an update.
04:36:56 <nyc> It doesn't increase the number of writes needed for an update.
04:37:33 <mrvn> nyc: it adds at least the root of the files b+tree to the depth.
04:38:10 <nyc> There is typically slack in the B+ tree for insertions and deletions.
04:38:22 <mrvn> might even out for large files because the inode b+ tree would be smaller. but small files would be deeper.
04:39:14 <nyc> Small files would be one extent and would most likely collapse entirely into the inode.
04:39:20 <mrvn> nyc: COW filesystem. you need to write every block along the path of the tree.
04:40:27 <nyc> COW fs is another matter entirely.
04:40:45 <mrvn> nyc: I'm not aware of any b+ tree based FS that isn't COW.
04:41:04 <nyc> JFS and XFS just journal.
04:41:21 <nyc> I think XFS is metadata-only journaling, even.
04:43:04 <nyc> JFS too.
04:43:15 <mrvn> are you sure xfs has b+trees for the metadata?
04:44:33 <nyc> xfs uses B+ trees for everything.
04:45:23 <mrvn> looks like it. At least wikipedia mentions free space and directories ebing b+trees.
04:45:35 <doug16k> ntfs uses b+ trees for directories and security data
04:46:25 <mrvn> I stand corrected
04:48:11 <mrvn> but as said journaling has it's own set of problems. Either way you have something far more complex than ext2 or fat.
04:49:19 <geist> most large COW file systems do the 'one largeass tree' thing
04:49:35 <geist> that lets you easily snapshot the system by just accumulating new pieces of the tree as you change it
04:50:10 <mrvn> works with trees of trees too but it just complicates things all around.
04:50:13 <geist> well, easy in concept. hard to pull off
04:50:21 <geist> but the idea is conceptually simple, yes
04:51:02 <geist> WAFL is the first hard core fs i know of that worked that way, but i suspect there were some fairly advanced stuff in the early 80s
04:51:17 <geist> spiralog was a VMS one that i think was log based as well
04:51:19 <mrvn> snapshoting is easy. Diffing two snapshots though is harder.
04:51:27 <nyc> VMS was flat out versioning.
04:51:45 <nyc> https://en.wikipedia.org/wiki/Files-11
04:52:00 <geist> yes, hat's files-11, but there was something called spiralog i think which was different
04:52:04 <geist> no idea if it was used much
04:52:24 <geist> mrvn: yes and no. i think it's fairly simple to take two versions of the same fs tree and compute the difference between them
04:52:31 <geist> it's extremely cheap with the btrfs diff tool
04:52:43 <geist> you give it two snapshots and the generations and it blats out the delta really fast
04:52:48 <mrvn> geist: the tree might have been rebalanced (root added or removed)
04:53:11 <geist> of course, but that still doesn't mean it's not already set up to diff the delta fairly easily
04:53:26 <geist> certainly much faster than having to iterate over every dir and inode in fs order
04:53:45 <mrvn> but that means you can't just compare the two roots, then compare each child against the other child.
04:53:48 <clever> geist: for comparison against zfs, every directory and file has a "birth tx" that includes the transaction# it was created in (and due to being immutable, changing a directory is just making a new dir that is similar)
04:53:56 <geist> especially since in practice most of it probably wont have been rebalanced that much, especially if you have a lot of fan out in your btrees
04:54:03 <nyc> https://pdfs.semanticscholar.org/acc9/e673228a16461db57c3b975355c68e7c4d2b.pdf <-- Spiralog was inspired by Sprite.
04:54:05 <geist> clever: right
04:54:15 <clever> geist: so, as you scan a directory, you can instantly know if its child-dirs are before, inside, or after, the diff window
04:54:24 <geist> exactly. i think btrfs may have something like that
04:54:30 <clever> and you only have to recurse down dirs that where modified within the window you want a diff on
04:54:32 <mrvn> I guess you can use the keys to decide wether you need to go down on the left or the right tree to find matching subtrees.
04:54:32 <geist> it certainly has a birth tx for the top level root node of the tree
04:54:40 <clever> incremental snapshot sends work similarly
04:54:50 <geist> i've found in practice that the btrfs tree diff util is almost instantaneous, so clearly it has some fairly fast way to delta two trees
04:55:34 <clever> amd/iohk@zfs-auto-snap_monthly-2019-02-01-00h00 77.4M 13.8G 4.96G
04:55:36 <clever> amd/iohk@zfs-auto-snap_hourly-2019-02-06-10h00 0B 13.4G 59.7M
04:55:41 <mrvn> geist: you can look at the first and last key in each block. Then advance to the children in the bigger block and repeat.
04:55:50 <clever> NAME USED REFER WRITTEN
04:56:19 <geist> but even if you walked every inode in the trees, you can quickly tell if their extents are identical or not
04:56:25 <clever> geist: i'm not entirely sure why, but a diff between these 2 nodes for ex, is fairly slow
04:56:33 <geist> since at least in btrfs, the extent nodes are all ref pointers basically
04:56:36 <clever> it first printed the root dir, and then has just gone nothing since
04:56:38 <mrvn> geist: bot inodes. the b+tree nodes.
04:57:01 <clever> after a delay, it has begun spitting out source files that where deleted or changed
04:57:05 <geist> but i suspect btrfs at least has a tx gen thing in the inode, as clever said
04:57:10 <geist> that would be super trivial to do
04:58:04 <mrvn> geist: shouldn't be needed. the keys should be enough.
04:58:42 <geist> probably? easy enough to find out
04:59:22 <clever> checking iostat, the diff i'm doing is not IO intensive
04:59:30 <geist> https://btrfs.wiki.kernel.org/index.php/Data_Structures#btrfs_inode_item the 'transid' is probably that
04:59:50 <mrvn> worst cae you go all the way to the left most child. then go up as far as the trees are same.
05:00:16 <mrvn> clever: rotating disks? maybe even spun down?
05:01:02 <mrvn> clever: is the second run faster?
05:01:20 <clever> mrvn: i think its cpu bound, top shows it at 94% usage
05:01:40 <clever> and its holding there even when not showing things
05:02:02 <clever> so its the diff that is cpu intensive, not the act of printing the results
05:02:58 <clever> ah, and its kernel usage, `time` says it spent 1.1 sec in userland, and 5m40s in kernel
05:03:07 <mrvn> clever: it's still worlds faster than actually diffing the files
05:03:16 <clever> yep
05:03:43 <clever> if i diff 2 snapshots with zero changes, its instant, since the root dirs are identical
05:03:49 <clever> it didnt bother with any recursion
05:03:51 <mrvn> I really like that you can pipe the diff through ssh and apply it on your backup server.
05:04:06 <clever> zfs diff doest show a file level diff
05:04:14 <nyc> I somehow got the West Australian Football League.
05:04:16 <clever> it just shows + - or M for each path
05:05:00 <mrvn> clever: resolving tree differences into file names might be costly.
05:05:42 <clever> if i take 2 snapshots with ~5mb of new data, the diff computes in a few secons
05:05:57 <clever> 7 on the 2nd run
05:06:15 <clever> and atotal of 541 files are modified
05:06:17 <kingoffrance> nyc: he probably meant https://en.wikipedia.org/wiki/List_of_log-structured_file_systems WAFL
05:06:19 <geist> nyc: from looking a bit closer, looks like WAFL is more like BSD LFS
05:06:43 <geist> https://en.wikipedia.org/wiki/Write_Anywhere_File_Layout
05:07:11 <nyc> I don't think COW really involves all data structures in the fs being COW'd.
05:07:25 <nyc> It looks like snapshots are used.
05:07:37 <geist> not necessarily, but snapshotting the fs generally involves it
05:08:01 <geist> but it doesn't necessarrily involve having to use a single large btree or anything
05:08:26 <clever> nyc: from what ive heard, zfs will COW every directory up to the root of a dataset, and then write the location of the new root to a ring-buffer
05:08:27 <geist> stuff like BSD LFS and looks like WAFL are more of a classic inode/dir/block pointer thing
05:08:38 <clever> so changing any file, involves changing every parent dir
05:09:34 <geist> right, and you ripple all the way to the top, and then periodically update the superblock to point at the new rroot
05:09:58 <geist> you need some sort of way of indirecting inode numbers, but otherwise the damage should be mitigated to just the inodes the way up
05:12:30 <geist> or maybe you dont indirect the inode numbers and they just change, so the dir they're contained in does, which changes the dir, etc etc
05:12:51 <geist> but you then batch all of that up and periodically toss down a new snapshot, and GC the old stuff. it's pretty clever
05:13:33 <nyc> It seems to me that things either want the most recent version of a thing or a particular version.
05:14:26 <geist> i suspect the thing that really makes all of these things usable now is the advent of flash storage
05:14:30 <nyc> So when you break off a snapshot at a change, it's only an old version that needs to be generated etc.
05:14:39 <geist> with spinny disks i think this sort of snapshotty thing generates terribly fragmented disks
05:14:46 <mrvn> clever: if you don't COW the metadata then what's the point?
05:16:09 <nyc> mrvn: I'll assume that was directed toward me. One would COW the metadata.
05:16:14 <mrvn> clever: also usualy the B+tree contains entries using "inode, file offset" -> "data segment". You ripple all through the tree but not through directories.
05:16:32 <mrvn> nyc: yes
05:17:21 <mrvn> nyc: I could see having COW for metadata and edit file data in place. But that's like metadata journaling. You get file corruption on crash
05:21:10 <mrvn> One trick modern B+ trees in filesystems use is to reserve a bit of space in every node to store updates. So first change to the B+ tree just records the change in the root node. When that is full the changes get pushed to the children. So you can collect a bunch of changes to leafs without having to ripple the whole tree every time.
05:22:59 <mrvn> probably using any free space in each node too. They might be only half full.
05:34:05 <nyc> I'm not finding the kseg boundaries by gdb'ing qemu.
05:34:49 <kingoffrance> actually, my "sparse restore disk image" may make more sense if one was restoring memory image to RAM...presumably after a reboot, things are "zeroed" ?
05:35:00 <kingoffrance> that might be a case where such would make sense
05:35:33 <doug16k> things aren't zeroed
05:35:45 <doug16k> in qemu and probably other vms they will be though
05:36:04 <doug16k> cold boot that is
05:37:38 <immibis> they may still contain data from before the reboot, but only some of it
05:38:02 <immibis> well after a reboot ram probably contains the same data. after a power cycle, maybe not
05:39:39 <fluffy0> i don't think you can ever reliably expect memory to be zeroed on startup
05:41:07 <doug16k> it would be interesting to fill all usable memory with a pattern and reboot, then scan and see which pages got overwritten
05:42:04 <doug16k> ...by the firmware of course
05:42:45 <doug16k> might be nice to put some critical data there to mitigate reboot attacks
06:32:15 <geist> mrvn: right, i think if you have very dense btree nodes with lots of fan out and you let an inner node get pretty sparse before rebalancing, you can avoid excessive rebalancing for trivial changes a lot
06:35:26 <nyc> Yeah, usually there are thresholds for rebalancing.
06:42:56 <nyc> I think a lot of implementations don't even bother doing real deletions in B+ trees.
06:44:14 <nyc> I'm big on capitalizing on range-based algorithms and therefore B+ trees. mrvn seems to be very anti- balanced tree on account of rebalancing "overhead."
06:44:50 <geist> it's all tradeoffs. there are always some downside to something, but you have to look at it relative to the upsides
06:46:42 <nyc> http://sci-hub.tw/10.1145/202660.202666
08:29:35 <kingoffrance> interestingly enough, i was reading about djb qhasm, which refers to knuth circa 1972 IIRC paper re: when to/not use goto, which mentions future languages might us indentation for control structure
08:29:40 <kingoffrance> wonder if that is where python got the idea
08:30:32 <immibis> python probably got it from a language that got it from a language that got it from a language that got it from that paper
08:30:38 <kingoffrance> i almost want the opposite of qhasm, a "portable" way to do non-math asm operations :/
08:30:48 <kingoffrance> not for speed, for boring stuff only done from asm
08:31:44 <nyc> Amoeba?
08:32:45 <klange> Python got it from ABC. https://en.wikipedia.org/wiki/ABC_(programming_language)
08:33:18 <immibis> kingoffrance: what like lgdt? that's not portable
08:38:42 <nyc> Well, just number temporaries of some small set of fundamental types and make 3-address code of some fundamental set of arithmetic operations.
08:39:40 <kingoffrance> knuth makes an interesting comment when he is adding up "time" for his pseudocode: paraphrase this is for hypothetical multiregister computer (what? there were non-multiple register computers? i wonder what he meant)
08:40:07 <klange> vs. a stack machine?
08:40:42 <kingoffrance> thats possible. or something like no register to register copies?
08:43:27 <kingoffrance> i used to have an old 1964 "computers and their uses" book i believe was name, talking about arithmetic instruction/hardware inside a single cpu/core. it was faster to "translate" to some weird base, so it could be parallelized. this was single cpu, but parallelization was quicker despite the overhead of translate -> do parallel arithmetic -> untranslate
08:43:55 <kingoffrance> even that single core/cpu system, still used "parallelization" internally for speed.
08:44:26 <nyc> Well, circuits always ran in parallel.
08:44:30 <Ameisen> kingoffrance - I'll bet the next paper was saying that the only proper indendention woudl be tabs
08:44:33 <Ameisen> then a response about spacfes.
08:44:36 <Ameisen> spaces*
08:44:43 <Ameisen> then one claiming only *vertical* tabs.
08:44:48 <nyc> wrt. portable asm why not llvm?
08:44:48 <Ameisen> I, of course, use zero-width spaces.
08:45:04 <Ameisen> llvm bitcode isn't completely portable.
08:46:03 <nyc> This is interesting: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.68.4273&rep=rep1&type=pdf
08:53:45 <geist> kingoffrance: of course. many machines were single register
08:53:55 <geist> that's a very common pattern, even up into the 80s
08:54:06 <geist> usually accumulator based. or memory to memory entirely
09:05:15 <mobile_c_> does anyone work with glibc dynamic linkers
09:07:40 <nyc`> I need to find the relevant qemu data structures.
09:12:18 <nyc`> The PC was one register.
09:16:52 <immibis> kingoffrance: carry save addition?
09:16:55 <immibis> https://en.wikipedia.org/wiki/Carry-save_adder
09:28:25 <kingoffrance> actually qhasm .q files arent "portable" either. it is supposed to still be "higher level" however
09:29:25 <kingoffrance> immibis: well, this was division IIRC. single word, multiple bytes. i just thought concept could apply to parallelize modern "big ints" (they may surely be faster parallel big int stuff nowadays)
09:31:48 <nyc`> How do they make it non-portable?
09:32:28 <kingoffrance> well, the .q files you can do pseudo-c-ish stuff, but still hardcoded reg names and such. so "Translation" to another cpu might be relatively simple
09:32:36 <kingoffrance> but i believe you still have to write one .q file per cpu
09:33:07 <nyc`> kingoffrance: They could do discrete Fourier transforms for multiplication and division.
09:34:37 <kingoffrance> that old book might have even been for BCD computer, dont recall (it mentioned various different types). that would explain why "normal" representation was so "slow"
09:36:28 <geist> look at something like a 6502. it has one general purpose register
09:36:33 <nyc`> BCD is faster in hardware than software.
09:36:41 <geist> and two index registers that are not GP
09:36:55 <geist> that's a fairly common arch for simple accumulator based machines
09:44:00 <nyc`> I should be able to fish for wherever the serial port went in qemu.
09:47:03 <immibis> i considered a zero-register relay computer, but that's only if you don't count a MAR, and MDR, and a few bits for in/out of a bitwise ALU
09:48:15 <mobile_c_> immibis: what hardware do you need to emulate for that 0.0
09:48:18 <nyc`> If you're going to go CISC you might as well go VAX.
09:49:24 <nyc`> I'll have to find a VAX emulator.
09:50:00 <immibis> mobile_c_: ?
09:51:07 <mobile_c_> zero-register relay computer
09:52:24 <nyc`> Are there no IA64 system emulators left?
09:55:47 <immibis> mobile_c_: well that was going to be my own hardware
10:00:02 <kingoffrance> immibis: i could be way off, mixing up knuth, but i coulda swore it was some kind of "negative base" or something? maybe that rings a bell? im sure there is better nowadays
10:00:20 <kingoffrance> logarithmic maybe?
10:01:04 <kingoffrance> the idea was convert to this weird base, then you can operate on each branch in parallel, then convert back
10:01:20 <immibis> are you sure it's not a carry-save adder?
10:03:56 <nyc`> I'm pretty sure it's a discrete FFT, not a base conversion.
10:15:19 <kingoffrance> immibis: went thru my old notes. "computers and their uses" is wrong book, old book by ibm guy, mentions he was worried about tracking back then
10:15:38 <kingoffrance> it is ""the logic of computer arithmetic" dont have a date. residue bases and multi-radix bases
10:16:49 <kingoffrance> but my notes say that did *not* work for divide (but you can implement using +-* of course) . that may still be wrong, but that is what my notes say :)
10:18:27 <kingoffrance> https://books.google.com/books/about/The_Logic_of_Computer_Arithmetic.html?id=24gmAAAAMAAJ that looks way too long a book, but that has got to be it, 1963. i just found it in a thrift store randomly
10:24:49 <immibis> aren't you glad we settled on binary?
10:26:50 <nyc`> Knuth's radix 2i numbers would have been nice.
10:30:05 <nyc`> Quaternions or geometric algebras might have been nice, too.
10:53:23 <Ameisen> base 1 computers.
10:54:32 <kingoffrance> apologies for rambling. at the time, i think knuth had a non-parallel "big int divisoin" is why i mentioned it. it looks well known nowadays e.g. cryptography books probably have many variations
10:55:16 <kingoffrance> ie knuth probably mentioned it too, but no pseudocode algorithm
10:56:18 <kingoffrance> and e.g. "hackers delight" other book i had with "big int division" may have mentioned it too, but no algorithm/pseudocode
10:56:18 <nyc`> Karutsuba or something like that?
10:56:40 <kingoffrance> i just wasnt reading the write books :/
10:57:10 <kingoffrance> and/or authors didnt write for "many cores" being "normal" :/
10:58:17 <SopaXorzTaker> kingoffrance, simple long division, but likely in base-2 or 256
10:58:24 <SopaXorzTaker> it's pretty simple actually
10:58:44 <nyc`> Discrete FFT algorithms parallelize well but aren't faster until the numbers are very large.
11:04:15 <nyc`> Karatsuba or whatever doesn't parallelize IIRC.
11:06:26 <nyc`> (This is mostly multiplication.)
11:11:15 <nyc`> I think number-theoretic functions would be nice to have as instructions, e.g. d(n), μ(n), φ(n), etc.
11:12:48 <nyc`> A few transcendental functions like ln(z), e^(z), sn(z,k), cn(z,k), dn(z,k), etc. too.
11:14:34 <nyc`> Jacobi and/or Legendre symbols, too, of course. And gcd.
11:16:03 <nyc`> It's really nice to have hardware acceleration for thinga that may well show up in inner loops.
11:16:48 <kingoffrance> e.g. if you just read https://en.wikipedia.org/wiki/Mixed_radix and are new, it makes passing mention to FFT, but zero mention of parallelization properties
11:26:04 <nyc`> kingoffrance: I think the digits in some radix are FFT'd number-theoretically.
11:27:20 <nyc`> kingoffrance: I'm foggy on the details, but am sure recent articles on arithmetic mention it.
11:29:14 <nyc`> kingoffrance: To be clear, though, gmp uses Karatsuba or some such instead because realistic multiple-precision integer usage isn't for big enough integers for the FFT algorithms to start winning yet.
11:32:05 <nyc`> I don't know the real cutoff point, though.
11:33:53 <nyc`> kingoffrance: The FFT itself is parallelizable, and once it's reduced to component-wise operations, it's very obvious how to parallelize.
11:53:31 <nyc`> kingoffrance: https://en.m.wikipedia.org/wiki/Multiplication_algorithm
11:54:08 <nyc`> kingoffrance: The details on FFT-based multiplication are there.
12:06:30 <FireFly> the inutition for fft-based multiplication for me is to look at multiplication of polynomials, and then see the correspondence between that and integers in some base
12:06:35 <mobile_c_> does anyone work with glibc dynamic linkers?
12:09:45 <klange> you gotta be more specific on the question; you're being too high-level, asking the wrong things
12:10:30 <mobile_c_> ok
12:10:31 <klange> what's going wrong when you load?
12:10:50 <mobile_c_> on dlopen does how does the glibc dynamic linker initialize a library
12:10:59 <nyc`> FireFly: I suppose I usually think of roots of unity.
12:11:53 <klange> Are you running ctors, init arrays, and init functions?
12:11:57 <mobile_c_> as it NEVER calls _dl_init nor call_init
12:13:50 <klange> how do you know it never calls those?
12:26:10 <mobile_c_> cus https://paste.pound-python.org/show/ddFCMrte9FL1G73n8OEy/
12:28:26 <mobile_c_> https://paste.pound-python.org/show/CAfkU9K7fGZ98s4fREx4/
12:28:28 <m3nt4L> Hi, I know i might be a little bit off topic, but i have a problem with C++, boost libs and unistd read. I am trying to use a standard read() call on an FD but for some reason, and only in some systems, this "conflicts" with the boost::asio::read. I am not usiing the boost asio namespace on that file. Could someone hint on how to solve this? Thanks in advance
12:31:11 <nyc`> I don't know that much about C++ and have never heard of boost libs.
12:33:22 <nyc`> I'm really smacking myself for taking so long to get serial IO going on MIPS. I hope it'll be easier for SPARC.
12:34:46 <m3nt4L> how s that? Saying MIPS means? You should have some serial controller to which you should point at. what is your platform?
12:35:04 <mobile_c_> https://paste.pound-python.org/show/CAfkU9K7fGZ98s4fREx4/ on dlopen does how does the glibc dynamic linker initialize a library? as it NEVER calls _dl_init nor call_init cus https://paste.pound-python.org/show/ddFCMrte9FL1G73n8OEy/ *
12:36:55 <nyc`> m3nt4L: I'm running on bare metal MIPS qemu with its ISA bus platform. I'll move on to the Malta when I can deal with PCI.
12:38:49 <nyc`> m3nt4L: qemu-system-mips64 -machine mips -serial stdio -kernel mips/hello
12:42:21 <nyc`> m3nt4L: I've got ARM and SPARC in progress, too. I can't rattle off asm off the top of my head for RISC-V, IBM POWER, Alpha, or OpenRISC, though.
12:43:39 <m3nt4L> nyc`, so basically the -serial stdio option is supposed to drive your stdio through a serial wrapper which doesnt work?
12:45:15 <nyc`> m3nt4L: I probably have the wrong address for the serial port in the IO space, or have the wrong range for IO space, or both.
12:45:23 <m3nt4L> nyc`, maybe you would find it usefull to actually drive the controller on your own. See how control & data of the controller works and try to directly write on it. It may be more painfull but with some small changes, you should be able to port it in other architectures as well
12:46:02 <m3nt4L> are you including this information on the linker
12:46:05 <m3nt4L> ?
12:46:10 <m3nt4L> sorry to ask, i havent done that
12:46:42 <nyc`> m3nt4L: I'm not sure what you mean. It's an emulated system and my code is running as the kernel.
12:47:40 <m3nt4L> nyc`, yes, still, there is some serial controller on the emulator, mapped in a specific physical memory range and you are trying to write there
12:48:11 <klange> mobile_c_: not sure how far I'd trust that, ld.so stuff can get weird
12:48:37 <nyc`> m3nt4L: Yes. Trick is finding the right address for it.
12:51:57 <nyc`> m3nt4L: It should be in kseg1 somewhere (uncached mapping of the lower 512MB of physical) and I have what's supposed to be the start of physical addresses reserved for device IO, but it's not at the usual PC offset into ISA IO space.
12:52:23 <nyc`> And it's supposed to be emulating ISA.
12:53:52 <nyc`> I may also be wrong about where kseg1 is in virtualspace.
12:58:35 <nyc`> m3nt4L: I'm mostly trying to fish the address out of qemu by gdb'ing it.
12:58:49 <m3nt4L> well the memory map should be available somewhere
01:00:40 <m3nt4L> The mipssim pseudo board emulation provides an environment similar to what the proprietary MIPS emulator uses for running Linux. It supports:
01:00:44 <m3nt4L> - PC style serial port
01:01:05 <nyc`> This can't be this difficult. I should probably just google for someone else's bare metal MIPS hello world.
01:01:26 <m3nt4L> thats what it says
01:01:58 <nyc`> That's what I'm stuck on for the moment, yes.
01:40:14 <nyc`> I wonder if my kernel is even loading.
02:12:09 <nyc`> After an hour gdb'ing qemu, it looks like it's loading, though maybe not at an ideal address.
02:16:21 <nyc`> It probably won't fix anything, but I should probably load it at a more normal address for a MIPS kernel.
02:23:29 <nyc`> I really hope SPARC, ARM, RISC-V, OpenRISC, Alpha, and IBM POWER aren't this difficult.
02:33:58 <painted> .theo
02:36:14 <nyc`> painted: What?
02:38:41 <lkurusa> .theo
02:38:44 <lkurusa> .bullshit
02:38:53 <lkurusa> glenda nooooo
02:38:58 <klange> RIP
02:39:10 <klange> nyc: bot that's supposed to be spitting out Theo de Raadt quotes
02:39:16 <klange> but is not here
02:39:25 <klange> I haven't seen glenda in a while.
02:39:26 <lkurusa> ... and generate bullshit
02:39:43 <lkurusa> like: "machine learning via evolution of vim"
02:40:20 <lkurusa> rumor is that this generator has spawned at least one startup that has received a $1m seed
02:41:04 <nyc> There's a quote of me from something like 1996 saying facetiously "MR. THE RAT: CAN YOU HELP ME INSTALL NETBSD? I WANT TO RUN INTERNET EXPLORER!"
02:41:35 <lkurusa> Ah, wow.
02:43:12 <nyc> It got a few chuckles out of people at the time.
02:43:35 <klange> Rumor is glenda was actually an artificial sentience trapped behind Theo de Raadt quotes and the .theo command, capable of speaking only when prompted.
02:43:56 <nyc> It was probably one of those "You had to be there" sorts of things.
02:44:21 <klange> An evaluation of the alleged source code says otherwise, but far too often were glenda's chosen quotes suspiciously relevant to the situation at hand.
02:44:51 <klange> But perhaps like Nostradamus it is merely a case of Theo's quotes being open to many interpretations.
02:45:37 <klange> Alas, it is late in the land of the rising sun, and I should sleep.
03:01:26 <lkurusa> good night klange
03:42:20 <nyc> PC = 0xffffffffbfc00000
03:43:51 <nyc> 0000000120000130 <__start>:
03:43:51 <nyc> 120000130: 240d0160 li t1,352
03:43:51 <nyc> 120000134: 3c0eb400 lui t2,0xb400
03:56:25 <nyc> It's spinning in addresses that aren't in my kernel.
05:15:59 <geist> nyc`: yep, that's why i was telling you to hard verify it at the qemu side
05:16:08 <geist> make sure it's actually linking and running the right address, etc
05:16:15 <geist> looks like there's some sort of linkage problem
05:20:08 <nyc`> I forgot to write a linker script.
05:28:38 <nyc`> I'm out at the doctor's office, so it may take a while to get back to where I can code.
05:57:02 <nyc`> qemu supports a lot more stuff than is packaged for Debian.
05:57:27 <nyc`> RISC-V and OpenRISC in particular.
05:59:16 <nyc`> IA64 looks hard to come by even in emulation, sadly. Alpha is there. VAX also looks tough to come by.
06:01:08 <geist> yep. vax you can use simh for
06:01:11 <geist> if you really want
06:01:24 <geist> simh is a great emulator for old machines
06:01:41 <geist> i actually have a netbsd image on simh right now, works pretty well. at least runs faster than the original machine
06:02:19 <geist> but, frankly, i think you may be getting a bit off track. what's your reason for being interested in these old machines?
06:02:49 <nyc`> Oh, it's just good to be portable.
06:03:12 <geist> well, it also makes some amount of sense to restrict the portability to things within a same set of features
06:03:22 <geist> ie, only 64bit. or only things with modern mmus, etc
06:03:39 <geist> or things that are only little endian, etc
06:04:41 <geist> being portable is nice, but being portable to obsolete things mayy be a bit too far
06:06:10 <geist> toolchain availability will also eventually hamper you. though vax is still obtainable
06:06:14 <geist> i think
06:09:53 <nyc`> I'm not 100% sure w
06:11:30 <nyc`> I'm not 100% sure about how far I can go.
06:13:23 <nyc`> I do hope I can scale both up and down. I just have to get to the point where I can even talk about generic code running across architectures.
06:14:34 <nyc`> The VAX would be a particularly good demonstration of some tricks I've got up my sleeve with 512B pages, though MIPS is probably an even better one.
06:17:55 <nyc`> IA64 has the broadest spectrum of page sizes, so it would really be the ultimate thing EOL or no.
06:18:03 <doug16k> imho, you can't write code that you can't actually test. do you have a real VAX on hand?
06:18:31 <doug16k> working in an emulator only proves so much. it still might not actually work on a real machine
06:19:36 <mawk> I have to code a lightweight qemu
06:19:38 <mawk> using KVM
06:20:02 <nyc`> The part that matters for my purposes is handling 512B pages.
06:20:45 <mawk> so I have this bzImage thing, according to the linux boot protocol I can load directly its 32 bits part is absence of a BIOS
06:20:58 <mawk> that means the 32-bit part of linux won't ever make BIOS calls and I won't have to write the BIOS interrupts ?
06:21:23 <mawk> in absence*
06:21:35 <mawk> the boot protocol: https://www.kernel.org/doc/Documentation/x86/boot.txt
06:22:11 <nyc`> x86-32 PAE would actually be another demonstration of the algorithms, though it would probably just be a 32-bit kernel on a 64-bit machine.
06:23:34 <nyc`> MIPS is appealing because it has 1KB page sizes while going up to 256MB pages.
06:23:55 <nyc`> In power of 4 steppings.
06:26:00 <geist> there's a particular architectype here
06:26:07 <geist> ie, hacking an os to prove some particular algorithm
06:26:37 <mawk> if it's an algorithm you don't need testing
06:26:42 <mawk> other than for contenting your mind
06:26:48 <mawk> you can analyze with a pen and a paper
06:26:53 <nyc`> Well, it's what people do for publishing papers all the time.
06:27:06 <mawk> yeah publication has high standards
06:27:22 <mawk> but it's not even systematic
06:27:31 <mawk> you can invent a general algorithm and not test it for various reasons
06:28:24 <nyc`> I don't think it does, people just ride on the backing from their institutional affiliations e.g. uni or employer.
06:30:20 <nyc`> I'd basically be trying to do a Terry Davis without being a racist anti-LGBT religious lunatic.
06:30:37 <mawk> lol
06:31:15 <nyc`> And actually be proving some algorithms worth publishing in the process.
06:31:43 <nyc`> And hopefully attracting some employers while I'm at it.
06:44:18 <knebulae> @nyc: Terry wrote a product no one would/could realistically use because he was a bipolar schizophrenic. He may have had the necessary skills to get published at one time, but I think your approach to this whole thing is decidedly different.
06:44:49 <nyc`> I'll probably have to do serious crash courses in a bunch of things in the kernel I don't know much about like fs's and the net stack.
06:45:33 <nyc`> knebulae: I'm also not on a mission from any gods.
06:45:37 <knebulae> @nyc`: you know I'm doing the same, just with the "formalities" (i.e. algorithmic analysis and mathematics).
06:46:12 <knebulae> If you want to be taken seriously, you have to be able to talk the talk.
07:05:27 <nyc`> Filesystem scalability is tricky stuff.
07:05:38 <nyc`> So is network.
07:07:00 <nyc`> I'm going to be in a world of hurt with fs's because I won't be able to borrow other kernels' code.
07:16:48 <nyc`> The design goal is worth the pain, though.
07:22:36 <nyc`> That's just a combination of aio and stacklessness at least wrt. user threads and processes.
07:23:55 <nyc`> Coremaplessness is also in the mix, though it's not as big of a deal for various reasons.
07:42:08 <mawk> what peripherals do linux need to boot ?
07:42:17 <mawk> I provide it with a serial port, and IRQ chip
07:43:10 <mawk> the irq chip business includes an ioapic
07:43:16 <nyc`> mawk: You may need a NIC for nfsroot or a disk.
07:43:30 <mawk> yeah NICs are just the icing on the cake
07:43:35 <mawk> I'm talking about really required peripherals
07:43:37 <mawk> for instance timers
07:44:06 <mawk> I'm using an initramfs so I don't need a disk or a NIC for now
07:44:07 <nyc`> Linux will just panic without a root filesystem.
07:44:24 <lkurusa> depends on your definition of boot to be fair
07:44:26 <mawk> the root is the initramfs itself
07:44:39 <nyc`> That might do, though initrd's suck.
07:44:53 <mawk> it's initramfs, not initrd
07:44:56 <mawk> it's the newer thing
07:45:23 <nyc`> It's pretty much the same.
07:45:30 <mawk> yeah in the end
07:45:33 <mawk> but the form is different
07:45:41 <mawk> the initrd is an ugly disk image, while the initramfs is a superior compressed image
07:54:55 <mawk> so, I don't need a timer ? the ioapic can do that itself ?
07:55:00 <mawk> and the tsc is provided by KVM already I guess
07:55:35 <mawk> and I can tell linux to stop looking for a hpet
08:35:41 <nyc> Hmm. The Linux linker scripts are difficult to interpret.
08:37:40 <Brnocrist> linker is hard
08:53:28 <nyc> Brnocrist: I think the Linux linker scripts are obfuscated.
08:56:53 <mawk> I tried to read the linker script for linux module
08:57:03 <mawk> to see how it constructs the __modinfo section and stuff
08:57:09 <mawk> your hypothesis may be correct
09:00:35 <Ameisen> So... if I tell clang to statically link glibc when building ruby... the ruby binary it builds segfaults.
09:00:39 <Ameisen> trivial stuff works fine
09:02:09 <mawk> I didn't know you could statically link glibc
09:02:19 <mawk> it makes a fake static build that dlopens the libc for some functions
09:02:24 <mawk> eg dns resolution
09:02:36 <mawk> so you succeeded in making a real static link ?
09:15:18 <nyc> I guess I had better link it to start at xkseg.
10:01:49 <nyc`> I created the . text.sched sections in Linux.
10:02:36 <nyc`> They were originally for a sort of profiling to see where context switches were coming from.
10:03:58 <nyc`> I ended up doing the stack unwinding code for my bunch of arches for get_wchan.
10:08:03 <nyc> They have a totally useless scheduler profiling thing in there that got put in instead of my original code.
10:08:33 <nyc> It just reports massive numbers of hits to __down and io_schedule() instead of anything useful.
11:23:54 <mobile_c_> "If a library has an .init section, the loader calls it to do library-specific initializations, such as C++ static constructors, and any .fini section is noted to be run at exit time."
11:24:02 <mobile_c_> why does that contradict glibc?
11:24:19 <nyc> ABI is incompatible with that of the selected emulation <-- ugh
11:25:09 <mobile_c_> https://paste.pound-python.org/show/uwhI6OGAyobEaJ4iOPFB/
11:29:18 <klange> mobile_c_: Where are you getting the quote from? It's wrong.
11:29:59 <klange> The loader does not call *sections*, it only does things with things in the dynamic table. DH_INIT_ARRAY, DH_INIT. These have gone in a few different sections throughout the history of gcc, .ctors, .init_array...
11:30:03 <mobile_c_> http://www.becbapatla.ac.in/cse/naveenv/docs/LL1.pdf
11:30:19 <klange> DT_*
11:30:33 <mobile_c_> [08:32] <ams> mobile_c_: there is a good book, linkers and loaders. read it. :)
11:31:21 <klange> diagrams hastily sketched on graph paper and scanned...
11:32:28 <klange> published 1999... yeah, this is going to be all sorts of outdated, and that line is particularly incorrect
11:32:39 <mobile_c_> x.x
11:35:38 <vdamewood> I have a copy of Linkers and Loaders. Most of it is still relevant, but it covers the IBM/360 (I think), and doesn't cover Mach-O binaries.
11:36:03 <mobile_c_> klange: what book do u recommend
11:36:16 <klange> None? Books are prone to rot.
11:36:21 <vdamewood> But what it says about ELF mostly covers Mach-O at a theoretical level, so Apple's docs can fill in the blanks.
11:36:23 <mobile_c_> x.x
11:36:24 <klange> I recommend pulling up the actual ELF spec.
11:43:12 <klange> At the most basic level, dynamic linking with elf is pretty straightforward. Load the libraries with the PHDRs, assigning base addresses as you go, keep a symbol table, perform relocations, call DT_INIT_ARRAY and DT_INIT... apparently the latter first...
11:43:54 <mobile_c_> ok
11:44:04 <mobile_c_> is DT_INIT meant to segfault when called?
11:44:35 <klange> The tough stuff is getting all the symbol table stuff and link ordering right, but that's not very important to most applications. It's not until you get into stuff call dlsym with RTLD_NEXT that things get interesting... and technically, POSIX software isn't supposed to use that.
11:44:39 <klange> brb
11:44:47 <klange> no, you missed a relocation or symbol resolution somewhere if you're segfaulting
11:47:06 <mobile_c_> ok
11:47:26 <mobile_c_> whats the smallest possible code that can produce an DT_INIT section
11:47:41 <mobile_c_> that can cause a DT_INIT to be produced*
11:48:20 <mobile_c_> as sending the readelf of glibc would probs be TLDR lol
11:50:12 <nyc> Now I've got the kernel where I think it should be loading in xkseg.
11:51:53 <nyc> Hmm. The paddr seems off, actually.
11:54:02 <nyc> Maybe it doesn't matter.
11:56:57 <nyc> Still an out-of-bounds PC.
11:57:29 <mobile_c_> klange: https://paste.pound-python.org/show/emWr0WkhOXO67o9twxmR/
11:59:16 <klange> this is way too much output and I have no idea what you want me to look at