channel logs for 2004 - 2010 are archived at http://tunes.org/~nef/logs/old/ ·· can't be searched
#osdev2 = #osdev @ Libera from 23may2021 to present
#osdev @ OPN/FreeNode from 3apr2001 to 23may2021
all other channels are on OPN/FreeNode from 2004 to present
http://bespin.org/~qz/search/?view=1&c=osdev&y=18&m=9&d=13
12:01:17 <klange> air :O
12:03:40 <air> lkurusa: thanks for the globbot info
04:00:27 <klange> https://youtu.be/ZMXT8r2-QhA
04:00:28 <bslsk05> 'ToaruOS-NIH - Syntax-highlighting line editor for shell' by K Lange (00:01:29)
04:04:22 <klys> klange, very colorful
04:04:50 <klys> https://www.intel.com/content/www/us/en/products/compare-products.html/intelproducts?productIds=97475,134857
04:04:52 <bslsk05> www.intel.com: Compare Products
05:45:14 <klange> added history and reverse search, patched in from the existing library... I think this is sufficient to be default for now.
05:45:34 <klange> Still so me missing key bindings, though, but it does tab completion and history and synatx highlighting and oversized lines and...
05:49:41 <geist> woot
05:50:00 <geist> i got our UEFI loader working on arm64 today. turns out it's pretty easy
05:50:09 <klange> neat
05:50:50 <geist> kind of as expected, you just compile for ARM and then it just works
06:05:10 <klys> joke https://pbs.twimg.com/media/DmbVC9QVsAA7P2G.jpg
06:09:40 <geist> hah
06:09:47 <klange> here, have a random demo video https://www.youtube.com/watch?v=7cJpMFfr0I8
06:09:48 <bslsk05> 'ToaruOS-NIH Demo Video (9/13)' by K Lange (00:04:06)
06:09:53 <klange> it's got shells and editors and pythons and stuff
06:10:19 <geist> klys: it's probably that new avx-512 unit they plugged on the side of newer xeons
06:10:25 <geist> cant trust it, it's an outsider
06:10:38 <tktech> klange, what are you writing your userspace tools in?
06:10:53 <klange> This is all C
06:12:02 <klys> geist did you compare the two xeons at the link listed above
06:12:26 <geist> what about?
06:12:36 <klys> which is better do you think
06:13:13 <geist> prolly the one on the right
06:14:15 <klys> it's also a fair bit cheaper according to wikipedia
06:14:35 <geist> the only real downside i've seen so far is the lower base frequency
06:15:22 <klys> it's https://en.wikipedia.org/wiki/Kaby_Lake vs. https://en.wikipedia.org/wiki/Coffee_Lake
06:15:22 <bslsk05> en.wikipedia.org: Kaby Lake - Wikipedia
06:15:23 <bslsk05> en.wikipedia.org: Coffee Lake - Wikipedia
06:15:29 <geist> otherwise it's basically either slightly better (more cores/cache, faster ram) or exactly identical
06:21:38 <geist> klys: what prices are you seeing on these?
06:23:33 <klys> wikipedia quotes 383 vs. 284. those are the best xeons I could find from Intel, and nobody is selling those models today.
06:24:38 <klys> if anybody was looking for a smaller process that's also not available.
06:25:02 <geist> the pricing unavailable thing is interesting. maybe they're only sold into certain channels?
06:25:19 <klys> yeah, interesting.
06:25:50 <klys> geist, what's your current intel testbed?
06:26:07 <geist> i have a few
06:26:09 <klys> klange, nice video
06:26:26 <klys> very responsive
06:26:29 <geist> for general hacking for zircon i have a few intel NUC,s generally 7000 series. i have a i3-6100 as well
06:26:49 <geist> though i dont boot it much, i have a haswell era workstation, dual socket
06:27:03 <geist> and for low end, i have an atom D330
06:27:19 <geist> and some sort of ivy bridge and and core2, though i dont use em much
06:27:29 <klys> what's your haswell part number?
06:27:34 <geist> my home server is a dual socket nehalem E5520, total work horse
06:27:53 <geist> E5-2690 v3
06:29:08 <geist> in AMD land i have a 1700x, 1500, 2700x, and a threadripper 1950x and a A10-7870K (kaveri era piledriver i think)
06:29:57 <klys> wikipedia quotes a $2090 release price on that part
06:30:05 <geist> probably
06:30:32 <klys> came out 8 sep 2014
06:30:33 <geist> it's a mid high end workstation class part
06:30:56 <geist> a new threadripper gives it a run for the money for sure
06:31:43 <geist> the haswell workstation is a few years old by now
06:31:54 <geist> they're starting to replace them with some newer hotness at work now, this is an older model
06:34:40 <geist> klys: note i didn't pay for the haswell. it's a work-provided workstation
06:35:06 <geist> if i were spending money now and wanted a beefy workstation machine, threadripper is by far the best bang for your buck
06:35:11 <klys> has it helped you with your work?
06:35:17 <geist> yah
06:35:25 <geist> but i compile big stuff all the time
06:35:57 <klys> so, the thing's different enough that you were able to debug cpu support for that kind of machine?
06:36:40 <klys> have you used it for baremetal tests, i mean?
06:36:46 <geist> depends. my experience is if you're doing osdev the cpu cores themselves aren't much different, though you should definitely try to run on a wide variety. intel vs amd is mandatory, some are different in subtle ways
06:37:10 <geist> for cpu support nehalem + is a continuum of newer features to support
06:37:21 <klys> well it would be interesting to see the AML data from that box
06:37:21 <geist> AVX, newer floating point stuff, SMAP/SMEP, etc
06:37:34 <geist> oh i can get that, in fact i was just fiddling with that ow
06:37:35 <klys> if just to see how much data there is
06:37:55 <geist> what i was just about to say is where things get complicated on the PCI side is when you go dual socket, or numa like (threadripper counts)
06:38:14 <geist> then you have multiple numa nodes and more importantly you end up with multiple PCI root bridges
06:38:43 <geist> klys: what d you want, the acpi dump? I have it saved up right here. the dual xeon, the threadripper (which is numa)
06:38:44 <klys> right, and you have an iommu that can isolate those two pci bridges
06:38:55 <geist> i can even toss in a dual socket ThunderX2, which is hilarious
06:39:02 <klys> sure, geist, I'll look at those
06:39:09 <geist> gimme a min
06:39:38 <geist> i actually went through and cached all this information
06:39:40 <klys> what's the name of the linux program for browsing ACPI data?
06:39:43 <geist> need to dig it out of a drive
06:40:00 <geist> if ou wantto dump stuff locally, then apt get acpica-tools
06:40:09 <geist> 'acpidump -b' will splat out a bunch of .bin files
06:40:27 <geist> then 'iasl -l *.dat' will decode them
06:40:30 <geist> s/bin/dat
06:40:43 <geist> i was *just* doing thaton an arm64 qemu instance to see what acpi looks like there
06:43:09 <klys> yesterday I got debian-riscv64 up and running with qemu.
06:43:19 <geist> cute. does it have an acpi dump as well?
06:45:26 <klys> says /sys/firmware/acpi is missing on riscv64.
06:45:35 <geist> ah
06:45:37 <klys> sortie, wb.
06:45:52 <sortie> Thank you
06:45:59 <klange> hey it's a sortie
06:45:59 <sortie> Took a little nap there
06:46:17 <sortie> Why not show up late for work and IRC in the morning?
06:46:42 <klange> i wanted to ask you to allow - in getopt option strings.
06:47:14 <klys> on this gen-2 lifebook, acpidump-acpica | wc -l; is 3186.
06:47:45 <geist> thats not very big
06:47:52 <klys> nope
06:48:44 <geist> 43985 on the dual xeon box
06:48:53 <klys> :)
06:49:23 <geist> btw the script i used o generate those dumps is at https://fuchsia.googlesource.com/zircon/+/master/scripts/linux-gather-system-info
06:49:23 <bslsk05> fuchsia.googlesource.com: scripts/linux-gather-system-info - zircon - Git at Google
06:49:39 <geist> just blats out some useful infos on stuff into a tarball
06:49:52 <sortie> klange: Like foo -abc-def?
06:50:10 <klange> yeah, it's a silly trick that lets regular getopt parse --foo
06:50:51 <klange> and it works in glibc, freebsd's libc, and macos's libc
06:50:59 <sortie> Ah and then you catch the - and handle subsequent options
06:51:05 <sortie> Reconstructing the long arg
06:51:14 <klange> well, you stick -: in your option string
06:52:23 <klange> so getopt does all the work for you and throws back - as the option char and "foo" as optarg
06:52:32 <sortie> Oh that's nice
06:52:48 <sortie> Maybe not as dirty as I thought
06:53:01 <sortie> Although if doesn't support arguments to that long option
06:53:02 <klange> while researching whether this was just a dirty hack, I learned some interesting things about spec-compliant getopt
06:53:18 <klange> like apparently posix is totally cool with you if you allow multibyte characters
06:53:27 <klange> but glibc doesn't implement that :(
06:53:28 <sortie> Yeah
06:53:38 <sortie> Kinda tricky because it returns an int
06:53:44 <sortie> Do I stuff the wchar_t in there? Guess I can
06:53:56 <klange> they specifically say "if it fits in an int" in whatever spec-y language they used
06:54:14 <klange> "Note that support for multi-byte option characters is only possible when such characters can be represented as type int."
06:54:16 <sortie> Can you file an issue on gitlab? I'd like to investigate what the remaining systems do with a few os-test testcases and to re-read POSIX carefully
06:54:58 <sortie> si-unit-parse -Å
06:55:32 <sortie> Unicode options is the only thing imagemagick needs to become a real wizard
06:55:51 <sortie> I'm sure it'll also make ffmpeg no harder to use
06:56:12 <klange> I was hoping it would just be hidden behind setting an appropriate locale in glibc; almost disappointed that they don't do it with all those other extensions :(
06:56:33 <sortie> I don't believe any system implements it
06:56:52 <sortie> It seems that getopt has always been about bytes and POSIX just tried to stuff in some unicode there while they were doing that anyway
06:57:04 <klange> I should implement it, but I need an IME or compose key handler first so I can type things... also better font support...
06:57:49 * geist just discovered the netboot ubuntu install images
06:58:00 <geist> very eas to spool something up on a virtual machine
06:59:34 <klys> geist, http://45.55.20.239/paste/system_info.Lifebook_T901.tar.bz2
06:59:56 <geist> pretty simple?
07:00:19 <klys> yeah
07:00:39 <geist> a little funny, sometimes laptops have more complex layouts
07:01:33 <klys> this one has vt-d and I have run virt-manager with windows 10 after booting over a serial line.
07:02:54 <klange> macOS actually has a bug with support `-` in that if you supply only a - it thinks its an option argument; there's also an obvious caveat here that it would cause -stuff-foo to be parsed as -s -t -u -f -f --foo
07:08:16 <klys> "cryptocurrency's 80-percent plunge is now worse than the dot-com crash" - slashdot.org
07:14:59 <klys> geist, where is the VT-d in cpuid.txt ?
07:15:11 <geist> not sure, honestly
07:15:33 <klys> k
07:15:41 <geist> iirc there are some other MSRs or whatnot that describe additional vmx stuff, i think
07:15:44 <geist> maybe it's there
07:21:50 <klys> your haswell box has no IOMMU it seems, the TR4 does, however, in the PCI listing.
07:22:05 <geist> may be disabled in the bios
07:22:34 <geist> yah linux days the haswell does
07:22:38 <klys> I have an AMD box here, socket AM3, which has no IOMMU, also
07:22:41 <geist> maybe it just doesn't get a PCI device like AMDs do
07:26:57 <nullp01nt3r> is there a way for disabling intel mgmt engine from kernelspace (ring 0)?
07:28:03 <klys> nullp01nt3r, have you searched around, say at slashdot.org and twitter.com for the normal way to do that?
07:30:04 <nullp01nt3r> yeah , i've searched across web, the only thing i found is reflashing the bios could disable intel me, but it is not feasible to flash bios, i think there must be some control path between x86 core and intel me chipset, but i couldn't found such information even though searched everywhere
07:32:36 <Mutabah> Tried the manuals?
07:37:23 <nullp01nt3r> yeah looked intel's manuals, but no luck
07:55:08 <klys> nullp01nter, have you read this stuff about that, http://blog.ptsecurity.com/2017/08/disabling-intel-me.html
07:55:09 <bslsk05> blog.ptsecurity.com: Positive Technologies - learn and secure : Disabling Intel ME 11 via undocumented mode
07:57:15 <klys> nullp01t3r*
07:57:29 <klys> nullp01nt3r*
08:02:11 <Mutabah> klys: tab is your friend :)
08:03:47 <klys> mutabah, how are you
08:08:59 <Mutabah> Not too bad, ticking along, thinking I should osdev again before remebering the hard problems I still ahve to solve :)
08:10:24 <klange> tab completion is great
08:10:51 <klys> yeah lyfe is full of those; I too am glad it's not tomorrow yet.
08:11:06 <klange> tomorrow is friday, I'm hoping it gets here quickly
08:11:45 <Mutabah> klange: amen
08:38:07 <sham1> klange: that line editor of yours reminds me of the fish shell
08:38:33 <sham1> With all of the syntax highlighting and such
08:39:45 <klange> The syntax highlighting is sort of a secondary thing I just got for free, I was mostly interested in getting something vaguely unicode-aware and capable of handling oversized lines, but it's a neat bonus
08:39:57 <sham1> Fair enough
08:40:11 <klange> Now I need to clean it all up.
09:46:28 <klange> And with some comments and a little bit of dead code removal... that's not half bad.
09:49:54 <lkurusa> klange: i suppose it will be on github eventually? :)
09:49:57 <lkurusa> would love to take a look
10:00:05 <lkurusa> 1
10:00:09 <lkurusa> oops
10:00:14 <klange> 2
10:02:33 <klange> lkurusa: yeah it's all up, lib/rline_exp.c is the where all the fun is
10:02:49 <lkurusa> great, thanks!
11:13:24 <klange> i just read the text of the EU Article 13 shit and freenode is now illegal
11:15:48 <glauxosdever> wow.
11:16:51 <Mutabah> Ooh, yeah, freenode would be large enough to fall into the "small-to-medium" loophole?
11:17:50 <klange> The text is too vague on who this applies to.
11:18:02 <lkurusa> it's not illegal
11:18:14 <klange> Is our little forum of 11 thousand users with a dozen posts a day providing access to "large amounts of works or other subject-matter uploaded by their users"?
11:18:16 <lkurusa> there's still an EP vote required before it's approved
11:18:25 <lkurusa> besides, it's only a _directive_ not a _regulation_
11:19:28 <lkurusa> directives are not enforced on the "federal" level, they need to be implemented by the states
11:19:35 <lkurusa> heck, they don't even _need_ to be implemented
11:19:41 <lkurusa> they can just choose to implement it
11:20:01 <lkurusa> i said it yesterday and i'll say it again, if you hold an EU passport call your MEPs :)
11:21:14 <glauxosdever> I'd also be worried about IMSLP
11:21:44 <Mutabah> klange: Well, since freenode doesn't keep logs internally, it might be ok?
11:22:02 <Mutabah> Depends on how it's phrased I guess. Freenode the IRC network is just a group of bouncers
11:22:22 <glauxosdever> It may be possible someone sends a copyrighted file to another person
11:22:23 <Mutabah> no hosting
11:22:30 <klange> depends on the interpreation of "providing access" - is ephemeral access sufficient? I mean you might even argue that once it's in everyone's IRC logs you can't remove it and thus it's impossible to comply.
11:24:28 <glauxosdever> I also have a thread on the forums about this -- it may be better to discuss this there too so more people are aware
11:31:04 <glauxosdever> Probably you have heard also about the pianist who played Bach and tried to upload his performance to facebook. Automatic filtering found that 43 seconds matched a performance copyrighted by Sony (a repeat offender, they also took down Sintel from youtube), so the performance was "infringing" copyrights
11:32:32 <klange> Yes, we already know exactly what these automated filtering systems do - we have YouTube as a prime example of how broken the idea is.
11:33:01 <klange> Now imagine trying to `git push` and your commits are rejected because they matched a bogus copyright claim from Oracle.
11:33:42 <glauxosdever> That would be equally stupid.
11:34:06 <glauxosdever> Hopefully git hosting services will be running on open source code
11:35:10 <glauxosdever> (So we can review what they are doing)
11:36:43 <klange> even if it's an open source codebase, the database of copyright matches will be closed off anyway
11:37:03 <klange> the only option will be to move to a smaller provider that's exempted by its size
11:37:20 <klange> this is going to erode large communities for the sake of not having to comply with the stupidity
11:37:29 <glauxosdever> Probably, or host your code yourself
11:37:49 <klange> yeah, but then I lose all of the community aspects, or have to provide them myself
11:38:18 <glauxosdever> Right. But git itself is decentralised
11:38:51 <klange> sure, but I don't use github and gitlab for their hosting, I use them for issue reports and so I can monitor forks and see people star things; I use them for discoverability
11:39:12 <glauxosdever> Sure, and this is where github and gitlab work better
11:39:27 <glauxosdever> But now with this stupidity from the EU.. I don't kknow
11:54:19 <nullp01nt3r> article 13 bs is actually showed us that people needs an open, free, decentralized, unregulated network, replacing internet
12:01:37 * asymptotically sends nullp01nt3r to a reeducation camp
12:22:27 <bauen1> isn't a database thats used to match copyright infringin content, copyright infringin if you don't have a license / permissions to use and distribute the database ?
12:25:54 <klange> It's usually some sort of hash or other derived content submitted by the purported copyright holder, not the actual content itself.
01:56:33 <lachlan_s> Where do the logs gets saved?
01:58:37 <klange> It's in the topic; new logs have moved to globbot
02:00:42 <lachlan_s> Thanks
02:07:29 <sham1> Is the bot GDPR-compliant?
02:26:53 <klys> ~
02:26:55 <klys> [3~
02:29:34 <klange> ... delete?
02:29:56 <klange> ^[[3~ is definitely delete
02:36:32 <lachlan_s> How did y'all get into osdev?
02:36:44 <lachlan_s> It's a pretty unusual hobby
02:38:52 <klange> In 2010, I wanted to take a course at my uni that did an OS project, as it was popular amongst my elder classmates, but I was denied because I had taken another course that conflicted with it credit-wise and the administration decided they didn't want people taking "useless" courses.
02:39:17 <klange> So I started working on my own project somewhat out of spite. I'd show them, dammit! I'd build a way better OS than anything that course could produce!
02:40:52 <klange> Spoilers: I totally did, but then I couldn't stop, it just kept going... now it's an addiction.
02:43:37 <lachlan_s> haha! You beat them
02:44:08 <lachlan_s> You ran the sigops group, right?
02:44:20 <klange> for like one semester
02:44:34 <lkurusa> lachlan_s: i found linux very interesting and wanted to build my own :)
02:44:45 <lachlan_s> A good a reason as any!
02:44:58 <lachlan_s> klange: I'm trying to organize a sigops chapter in the local acm group, not sure how to go about it
02:45:51 <klange> most of the SIGs don't officially do student chapters, but as an authorized ACM chapter you have the right to use the SIG names
02:46:04 <klange> some of them do, though, which can be confusing
02:49:54 <lachlan_s> Hmm, that's interesting.
02:49:59 <lkurusa> i don't know about your schools, but at my alma mater most of my peers hated operating systems
02:50:02 <lkurusa> Sadly
02:50:15 <lkurusa> it was difficult to find enough people to have a worthwhile conversation about systems papers
02:50:36 <lachlan_s> I'm thinking it's the same way here.
02:50:51 <lkurusa> at least between undergrads, graduate students were much better
02:51:01 <lachlan_s> I've been told several times that the work I do "is very much down in the weeds"
02:51:38 <lachlan_s> I wonder if the grad students here have a similar group.
02:52:27 <lkurusa> i found it easier to work with grad students than undergrads; graduate students wanted to accomplish something
02:52:35 <lkurusa> most of my undergrad peers wanted to "pass uni"
02:52:49 <lachlan_s> How did you get in contact with like-minded grads?
02:52:51 <lkurusa> and this is a top-10 worldwide university so meh
02:53:10 <lkurusa> lachlan_s: i talked to professors and they introduced me
02:53:33 <lkurusa> did some undergraduate research, word quickly goes around
02:55:43 <lachlan_s> Hmmm, I'll give that a try
02:55:53 <Shockk> so I'm implementing my stdio functions for my own libc, but I'm having a problem with printf("this should have a newline\n");
02:56:30 <Shockk> it works with -fno-builtin, but otherwise, gcc is replacing it with a call to puts and it doesn't print out a newline
02:56:33 <lachlan_s> I think I pissed off the professor who does os stuff here somehow, but I'll try talking to him again
02:56:41 <Shockk> but when I use puts myself, it does print a newline
02:57:26 <lkurusa> lachlan_s: haha, you'll be fine
02:57:51 <lachlan_s> My current comp sci professor is pretty awful, so I doubt she'll be much help here
02:57:56 <Shockk> so it's like gcc is inserting its own builtin puts that breaks it
03:01:07 <lkurusa> Shockk: check the disassembly
03:03:05 <Shockk> it looks like it's just calling my puts actually
03:03:19 <Shockk> hm
03:03:32 <Shockk> I have printf("line 1\n"); puts("line 2\n");
03:03:41 <Shockk> in the disassembly I see two calls to puts in there
03:03:54 <Shockk> however when I actually run this, the \n only appears for the explicit puts call
03:04:04 <Shockk> so I see line 1line 2\n
03:04:29 <lkurusa> are you sure your printf isn't cutting off the last character?
03:04:55 <Shockk> I'm sure, because if I add -fno-builtin, this works perfectly
03:05:09 <Shockk> like, gcc is replacing my printf("line 1\n"); with a puts
03:05:21 <Shockk> which for some reason doesn't print a newline though
03:05:48 <Stary> puts is supposed to add a newline, afaik
03:06:08 <lachlan_s> Yeah
03:06:45 <lachlan_s> It's removing the newline at the end of the string because it thinks your puts will add a newline
03:06:51 <Shockk> huh? I'm looking at the c standard and it just says "writes the string pointed to by s to the stream pointed to by stream. The terminating null character is not written."
03:07:05 <Shockk> oh wait
03:07:11 <Shockk> no, that's fputs
03:07:20 <Shockk> puts does append a newline, that's interesting
03:09:37 <lachlan_s> What's the best way for puts to add a newline without reallocating the string?
03:10:11 <lachlan_s> I suppose it could overwrite the null-terminator, but you have to be careful there not to write into const storage
03:10:13 <Mutabah> lachlan_s: call putc at the end?
03:10:22 <Shockk> right now I'm calling fputs, I was just going to additionally add a putc
03:10:24 <lachlan_s> Oh, or that, lol
03:10:55 <Shockk> I need to make sure to check both calls for errors though when determining what value to return
03:16:41 <Shockk> OH I get what was happening
03:16:59 <Shockk> it was replacing the printf call with a \n at the end, with a puts call *without* a \n
03:17:08 <Shockk> so now that I fixed puts, it does work
03:17:09 <lachlan_s> Yep
03:17:23 <lachlan_s> > It's removing the newline at the end of the string because it thinks your puts will add a newline
03:17:32 <Shockk> that's just something I glossed over because I expected fblah to behave like blah(stdout)
03:17:38 <Shockk> it seems to do that everywhere else
03:18:25 <Shockk> well now I have real output from my libc, it feels really good
03:18:35 <Shockk> no more newlib anywhere
03:44:36 <lachlan_s> Did any of you have to evacuate from hurricane florence?
03:44:45 <lachlan_s> It was quite a large swath of the states
04:51:26 <Shockk> I have a quick question
04:52:03 <Shockk> the C standard specifies that for memmove, the characters from s2 are first copied into a temporary array that does not overlap s1 or s2, and then those are copied into s1
04:52:31 <bcos_> I does?
04:52:34 <Shockk> is it permitted, though, to implement memmove simply by copying either forwards or backwards depending on what kind of overlap there is?
04:52:36 <Shockk> it does yes
04:53:20 <Shockk> but I see a clear way to implement it without overlap by just checking which of s1 and s2 come first in memory
04:53:31 <Shockk> (and either copying forwards or backwards)
04:53:39 <bcos_> I'd assume it actually says "memmove behaves as if characters from s2 are first copied into a temporary array, regardless of whether that's the acutally case or not"
04:53:58 <bcos_> *that's actually the case or not
04:54:18 <Shockk> hmm oh
04:54:35 <Shockk> it does say as if
04:55:07 <bcos_> It is permitted to do whatever you like as long as the behaviour is the same; including deciding to copy bytes forward or backward, and including copying significantly larger pieces at a time
04:55:10 <_mjg> the standard never tells you /how/ to do something
04:55:16 <_mjg> it tells you waht the effect is
04:55:34 <Shockk> hmm I see, okay
04:55:41 <_mjg> i have never seen a memmove which uses a tmp buffer
04:56:04 <Shockk> that makes sense
04:56:05 <Shockk> thanks
04:57:21 <bcos_> I'd probably start with some kind of "cmp s1, s2; jb go_forward; je do_nothing; ja go_backward"
05:00:24 <bcos_> ..followed by a test for overlapping that calls "memcpy()" when there isn't any overlapping
05:02:44 <bcos_> ..and then maybe an "if( (s1 | s1 | size) & 3 == 0) " test to determine if I can do whole 32-bit instead of bytes
05:25:42 <Shockk> hmm completely different question, but how exactly do I return a value back from a syscall?
05:25:52 <Shockk> can I just push something onto the stack before my `iret`?
05:26:18 <bcos_> Most people just return values in registers
05:26:24 <Shockk> oh right
05:26:32 <Shockk> okay that's a better idea
05:26:34 <Shockk> thanks
05:26:41 <sham1> You could also put stuff onto the stack
05:28:05 <bcos_> Can, it's just complicated/slow because the user-space stack isn't the kernel stack (and might not even be valid - you'd have to check in case user-space did "esp = address of kernel data" before calling kernel)
05:28:55 <Shockk> ah that's true
05:29:05 <Shockk> I think registers will work just fine for me
05:29:14 <_mjg> Shockk: the 'gold standard' is retval in rax + carry flag to indicate an error
05:29:20 <_mjg> if going for unix-like os i mean
05:29:31 <Shockk> right
05:29:42 <_mjg> i.e. if carry is set, shit failed and the value in eax is the errno
05:29:55 <Shockk> I confess I actually forgot how return values are even done for regular C functions lol
05:30:06 <Shockk> I'll figure it out
05:31:05 <sham1> On a C function implemented in Assembly you ofc follow the ABI
05:31:18 <sham1> For instance with UNIX-styled OSes that would be the System V
05:32:02 <Shockk> hmm oh it's just cdecl I'm wondering about, and apparently that just uses eax as well
05:32:04 <Shockk> that's handy
05:33:26 <bcos_> EAX or EDX:EAX or ST0 or it's all lies and the caller passes a pointer to a structure; depending on what the return type is
05:33:40 <Shockk> oh huh
05:35:22 <bcos_> (e.g. if you typedef a large structure type called "myStructType" and do "myStructType foo(int bar)" then it's essentially the same as "void foo(myStructType *temp, int bar)")
05:35:59 <Shockk> ah that's interesting
05:36:27 <bcos_> For small structures that can fit in EAX it's different I think
05:36:29 <sham1> Hmm, very interesting indeed
05:38:37 <bcos_> Sadly, most kernel functions should probably return multiple things (e.g. "status plus bytes sent/received", or "status plus file name" or ...) so it doesn't have to diddle with language specific nonsense (e.g. errno) and can let the language's standard library dead with language specific stuff; and that ends up being horrible for C
05:50:19 <lachlan_s> It's a shame that rich enums never got into the systemv abi
05:51:06 <zid> It's a shame C doesn't have real enum types
05:51:23 <sham1> What do you mean by "real enum types"
05:51:44 <zid> void f(enum BANANA *b, enum STRAWBERRY c){ *b = c; }
05:51:53 <zid> perfectly legal semantically, but probably shouldn't be
05:51:54 <sham1> Ah
05:52:16 <Shockk> strongly typed enums
05:52:27 <zid> afaik gcc at least can at least detect you using switch(enum_dude) { case MEMBER_FROM_A_DIFFERENT_ENUM:
05:52:28 <Shockk> like enum class from C++
05:52:54 <lachlan_s> the c type system is awful
05:52:56 <zid> I can just use a struct if I want, but that's just ugly
05:53:22 <sham1> Well C is a portable assembly
05:53:26 <lachlan_s> no
05:53:35 <lachlan_s> It's not
05:53:46 <zid> It's more like, a specific representation of like 8 different assemblies from 1980 :P
05:54:05 <lachlan_s> Not really, c is quite high level
05:54:37 <zid> Not particularly, it has some loose type checking and everything is "+ does exactly what every current cpu does with their add opcode"
05:54:49 <zid> basically every part of the design is driven by factors like that
05:55:33 <sham1> While it is high-level (especially since we aren't running PDP-11s) it still has a lot of assembly-like qualities in its designe
05:55:47 <zid> 7 of these contemporary machines have 1<<32 = 0, 1 of them has 1<<32 = 1 turned into "The result of the left shift-operator where the size of the right operand is the same as the bit-width of the type is implementation-defined..."
05:56:16 <zid> you can see that allll through the spec, it basically codifies the average of every cpu available in the 80s :P
05:56:30 <zid> anything they all did is defined, anything they differed on is IDH
05:56:46 <zid> a modern C would look vastly different just because it wouldn't be impacted by what a pdp-11 does with its shift opcode
05:57:06 <lachlan_s> Yeah, it's really up for debate.
05:57:27 <lachlan_s> It's design is definitely influenced by the behaviors of the machines it was designed to run on
05:57:32 <lachlan_s> s/it's/its
05:57:44 <zid> I'm like, 99% fine with C though, the only things I would change are mainly language things not '<< works in a stupid way' things
05:57:58 <sham1> Basically make stuff less stupid
05:57:59 <zid> like dereferencing pointers to structs with incmplete types until the first incomplete member
05:58:08 <zid> or enum types not being compatible
05:58:21 <zid> delete every function that starts with a w from the standard library
05:59:56 <zid> Although I am sure if you designed it today a lot of implicit conversions might target long instead of int, 64 seems pretty ubiq. these days
06:00:52 <zid> (windows can fuck off with its 32bit long too)
06:01:06 <sham1> Well Windows can do that in general :P
06:01:38 <zid> I get why they did it, but that's their mistake for putting LPLONG into their prototypes everywhere etc
06:02:36 <Shockk> ugh, for fgets do I have to read character by character in case I hit a newline?
06:02:41 <sham1> I'd probably do stuff like making memcpy and memmove use pointers to uint8_t instead of char
06:02:59 <zid> Shockk: Unless you're 'allowed' to read too much and scan for the \n sure
06:03:26 <zid> like if you're in buffered mode you might want to just read a kilobyte in advance and use an sse mess to scan 16 chars at a time for an '\n' or something
06:03:42 <Shockk> hm I can't do that yet but that sounds like a nice idea later
06:03:50 <zid> also fgets' design is stupid it shouldn't return the \n
06:04:26 <zid> standard library could use a few minor changes to align more with what posix ended up doing, a lot of the times posix is just C + experience
06:15:17 <lkurusa> there was a paper sometime ago that examined the influence of C on ISA design
06:15:21 <lkurusa> i'll see if i can find it
06:15:42 <Icefoz> I'd love to see that.
06:16:06 <sham1> I think it had stuff like how many CPUs have these things for doing flat memory stuff
06:16:25 <sham1> Even if it isn't really native
06:16:27 <lkurusa> the beauty of academia: too many papers in zotero :-)
06:16:33 <lkurusa> let me brew some tea and i'll find it
06:19:19 <lachlan_s> klange: You were at uiuc, right?
06:26:23 <lkurusa> https://ieeexplore.ieee.org/document/4051189/ This is the closest one i can find in my zotero but there's ought to be another paper that I can't find
06:26:26 <bslsk05> ieeexplore.ieee.org: Influence of Programming Techniques on the Design of Computers - IEEE Journals & Magazine
06:35:11 <geist> lkurusa: indeed. having a single stack too, i'd assume
06:35:24 <geist> or generally not being stack based
06:46:27 <sham1> In a twisted way I'd love to see the alternative universe where Forth became the language to influence all others
06:46:46 <zid> sick bastard
06:53:47 <geist> unrelated note: https://www.youtube.com/watch?v=wxL4ElboiuA
06:53:47 <bslsk05> 'The Art of Making a Nixie Tube' by Dalibor Farný (00:37:50)
06:53:59 <geist> relaxing watching someone build a complicated thing from scratch
06:54:05 <zid> The valve video is the best
06:54:07 <zid> it has good music too
06:54:22 <geist> oh yeah? do link
06:54:31 <zid> https://www.youtube.com/watch?v=EzyXMEpq4qw
06:54:50 <geist> ah fun
06:57:30 <zid> THE FOOT! :o
07:03:11 <geist> whos da foot
07:03:19 <zid> le queusot
07:03:22 <zid> translate that one for me
07:03:35 <geist> reminds me of The Elephants Foot
07:03:41 <geist> look it up, it's creepy as heck
07:03:47 <zid> I don't need to look it up
07:03:54 <zid> it's a slagged reactor
07:04:34 <zid> It's amazing it wasn't worse given how they basically did everything possible wrong
07:04:39 <geist> yah
07:04:57 <geist> apparently it's cooled off enough that don't immediately get a letal dose if you're in the same room
07:05:02 <geist> but you get like 30 seconds or something
07:05:29 <zid> reactor vessel made out of fammable material, nuclear is 'secret' so the operators don't know what to do, running tests under stupid circumstances, very slowly inserting control rods with large tips that displace a lot of water, etc
07:05:39 <zid> flammable*
07:06:07 <Icefoz> In like 200 million years when humans are gone and there's a new intelligent species running Earth
07:06:21 <Icefoz> there's gonna be a lot of very confused geologists for a while.
07:06:43 <zid> fun fact, carbon dating doesn't work after 1940 because of nuclear incidents
07:07:24 <geist> i've heard of that. there is a tiny bit of radioactive stuff spread everywhere that fouls it up
07:07:38 <zid> and they have to salvage pre ww2 ships for steel without radioactive carbon in it for making nuclear imaging equipment because it's less expensive than making it with carbon they had to purify
07:07:57 <geist> the Yamato will fly again!
07:08:09 <zid> yea, carbon dating just looks at the ratios between carbon isotopes, the more that's decayed the older it is
07:08:12 <geist> Space Battleship Yamato!
07:08:14 <zid> the ratio is fucked for anything after 1940s
07:10:02 <Icefoz> It still works, you just have to correct for the amount of radioactive stuff.
07:10:31 <Icefoz> Which you still have to do anyway 'cause the amount of radioactive carbon created through time varies.
07:10:36 <zid> Icefoz: except you can't
07:10:41 <zid> because you have no idea how much it was exposed to
07:10:49 <Icefoz> Sure you can.
07:10:52 <zid> was it sat in pripyat for 20 minutes? was it in a lead box at the time?
07:11:15 <Icefoz> Collect books from all over the world published in 1946 and analyze them.
07:11:19 <zid> what
07:11:20 <Icefoz> Then make a good guess.
07:11:41 <Icefoz> Hang on, I'm misunderstanding. One sec.
07:11:41 <zid> Except a 'good guess' is going to give you billions of years of error bars
07:11:59 <Icefoz> That's gonna be hard when the half-life of C13 is 14000 years.
07:12:00 <zid> because you need to know PRECISELY how much it was exposed during its entire history
07:12:14 <zid> half life, not full life
07:12:45 <moondeck> hi
07:12:46 <zid> you measure how much there SHOULD be, vs how much you have left. If you have 1/128th of what you expected it's 14000*n old
07:13:01 <Icefoz> I know.
07:13:06 <zid> but we're talking like, nanograms of material here
07:13:24 <Icefoz> Right, so after a point you don't use carbon dating 'cause there's so little source material left.
07:13:28 <zid> an extra nanogram or two is going to completely ruin the difference between 1/65536 and 1/4
07:13:32 <Icefoz> You use something with a longer halflife.
07:13:53 <Icefoz> That's the case for any radioactive dating technique.
07:14:04 <Icefoz> So is contamination. the Hard Part of it is trying to account for contamination.
07:14:13 <zid> Right, and I am saying you can't
07:14:19 <zid> I mean, you can, but your error bars would be huge
07:14:39 <Icefoz> I sorta doubt it.
07:14:52 <Shockk> hmm, if I invoke an interrupt to syscall into my kernel, how exactly should I implement some sort of blocking until the syscall is ready to return?
07:14:56 <Icefoz> If you take carbon samples from all over the world, all since 1946, you're going to have a general average and a few spikes.
07:15:09 <zid> Icefoz: How do you know if this skeleton from the 80s you're looking at ate exclusively pears from ukraine, vs wheat from the US?
07:15:09 <Shockk> i.e. I want to read a character from stdin via the syscall, but I want it to wait until one is available
07:15:16 <zid> You're going to get VASTLY different amount of contamination
07:15:41 <Icefoz> zid: You look at a lot of other skeletons in the same area.
07:15:45 <zid> Shockk: Schedule away from it.
07:15:50 <zid> Icefoz: ...
07:15:50 <Shockk> if I for(;;) { __asm__ ("hlt"); } during the syscall handler, it blocks keyboard interrupts from being received, it seems
07:15:58 <Icefoz> Sampling bias is a part of any physical science.
07:16:15 <zid> That's like saying you should measure the BMI of my home-town to figure out how much I weigh
07:16:31 <Icefoz> zid: If you have no other information to go on, that's not a bad idea.
07:16:43 <zid> It's a terrible idea
07:17:03 <Icefoz> It's an idea, and collecting data and doing statistics can tell you whether or not it's terrible.
07:17:30 <Icefoz> Instead of making gross generalizations because something obviously is or obviously is not wrong.
07:17:30 <geist> Shockk: interrupts are probably disabled. generally speaking cli;hlt is regarded as a full stop situation on the cpu, many emulators stop emulating if they see that
07:17:37 <geist> technically it isn't, because a NMI can still break it out of that
07:18:06 <moondeck> Where can i find some specification about setting graphic modes on x86?
07:18:13 <Shockk> hmm oh wait, I might disable interrupts when I begin handling the interrupt, need to check
07:18:38 <geist> Shockk: exactly. also if your syscall is coming in via the int vector, then depending on how you set up the IDT interrupts may be disabled on the way in
07:18:43 <geist> ie, if you use an int gate instead of a trap gate
07:19:02 <Shockk> hmm, I don't call cli so they should be enabled; so it must be the latter
07:19:21 <geist> but in general the first part of a syscall on the kernel side is pretty important to have ints disabled, so usually you enter the kernel with ints disabled, do some critical work to get things in the right place, and then reenable them
07:19:25 <geist> same with syscall/sysenter
07:19:37 <Shockk> oh really?
07:19:50 <Shockk> what kind of stuff can break if I don't initially disable interrupts?
07:19:50 <geist> well, assuming you have a reentrant, preemptive kernel
07:20:12 <geist> it depends. usually one thing will force you to require it
07:20:22 <geist> for example, on x86-64 you probably want to do a swapgs right off the bat
07:20:44 <geist> and you need to make sure that happens atomically on interrupt entry, prior to any interrupts coming along and interrupting that
07:20:57 <zid> Or you don't because of a stupid x86 hardware exploit I know of :P
07:21:22 <geist> in the case of syscall/sysret i believe there's a one or two cycle window there that you have to disaable ints to get stuff in the right place
07:21:32 <Shockk> hmm ah right
07:21:33 <geist> it's fairly common to need to do that
07:22:08 <geist> syscall is a pretty good example, because iirc (i may be confusing it again with sysenter) the SP is not swapped
07:22:26 <geist> so the first thing the kernel needs to do is swapgs to get gs: pointing at kernel state and then recover the kernel stack pointer
07:22:34 <geist> so you absolutely want that to be done without any interruption
07:22:49 <Shockk> this is going to take quite a bit more learning and research, right now though my pizza just got delivered
07:22:50 <geist> yah, that's syscall. sysenter does swap the SP, but doesn't save the user one
07:23:00 <geist> oh dang yes. pizza > *
07:23:31 <Shockk> you could say I scheduled it to yield at my front door
07:23:54 <moondeck> Also, are legacy peripherals specified in the intel manuals?
07:24:06 <zid> Fun exploit: mov ss can cause pending interrupts to be delayed, so the sequence mov ss, [rax]; int3/syscall can cause you to gain ring0's gs value in usermode if you set a breakpoint on the address in rax :)
07:24:13 <moondeck> Havent seen them in there, but havent really looked either.
07:24:31 <zid> because the cpu does mov ss, [rax]; syscall; swapgs; *interrupt handler for breakpoint*
08:05:57 <jjuran> Pizza: The highest-priority interrupt.®
08:06:55 <Shockk> most certainly
08:07:04 <Shockk> hm so it looks like I have to read up on processes and threads and everything
08:07:06 <Shockk> :(
08:09:17 <Shockk> so, I'm struggling to understand the basic principle of what's done when you block on an input stream
08:10:56 <Shockk> if for example I call syscall_get to retrieve a single character from stdin, and there's currently no input available, what exactly should kernel space be doing in the syscall handler?
08:11:24 <bcos_> When you block on anything, scheduler is told not to give thread CPU time. Then when whatever you're blocked waiting for happens kernel tells scheduler to let the thread have CPU time again
08:12:14 <Shockk> hmm, so I mean, should that thread just be waiting in kernel space in the interrupt handler for the syscall, in a hlt loop with some condition?
08:12:21 <_mjg> do you know what 'wait channels' are? (or perhaps condvars)
08:12:45 <Shockk> I've heard of them in passing
08:13:01 <_mjg> you put yourself to sleep waiting for an explicit wakeup, and the scheduler ignores you or 'forgets' that you exist
08:13:31 <bcos_> Should be no need for "hlt loop" because you're not getting any CPU time anyway
08:13:41 <Shockk> hmm
08:14:04 <_mjg> i think your question are a little bit too basic to start doing kernel work
08:14:16 <_mjg> have you considered doing something else like e.g. reimlementing libc?
08:14:22 <_mjg> (or whatever flows your boat)
08:14:49 <Shockk> I'm doing that right now too; but the kernel is what I'm really interested in
08:15:01 <_mjg> (floats?)
08:15:06 <bcos_> Sure, start with the locales/internationalisation part of libc! ;-)
08:15:08 <Shockk> doubles
08:15:11 <_mjg> seriously, my english is deterorating each year
08:15:13 <Shockk> I'd like double boats
08:15:42 <Shockk> I think I'm asking the wrong questions, due to my lack of knowledge of how exactly tasks and scheduling work
08:15:51 <Shockk> that's probably where I should start researching
08:16:11 <bcos_> Could maybe try this: https://wiki.osdev.org/Brendan%27s_Multi-tasking_Tutorial
08:16:13 <bslsk05> wiki.osdev.org: Brendan's Multi-tasking Tutorial - OSDev Wiki
08:16:25 <Shockk> I'll take a look
08:16:41 <bcos_> :-)
08:18:15 <Shockk> just to clarify something, monotasking systems have no scheduler as they only have a single thread of execution running?
08:19:13 * bcos_ nods
08:19:22 <Shockk> okay thanks
08:45:10 <Shockk> apologies for the excessive questions about this, I'm just trying to make sure I understand this as I go:
08:46:15 <Shockk> am I right in thinking that, if the cpu is currently in userspace for some particular process, then the way scheduling works on that is, a timer interrupt interrupts it in and the state of that task is saved so it can be restored later and continue where it left off?
08:50:09 <lnnb> https://en.wikipedia.org/wiki/Context_switch
08:50:09 <bslsk05> en.wikipedia.org: Context switch - Wikipedia
08:53:30 <Shockk> ahh I see now, that does mention timer interrupts
08:53:32 <Shockk> thanks
10:54:35 <Shockk> hmm so I'm looking at this: https://wiki.osdev.org/Multitasking_Systems#Kernel_Stack_Per_Task
10:54:37 <bslsk05> wiki.osdev.org: Multitasking Systems - OSDev Wiki
10:55:59 <Shockk> does this assume that the kernel is mapped to the same place for each virtual address space? I'm just asking to make sure I understand, because if it wasn't mapped to the same place, I imagine that doing mov cr3,eax may result in the next instruction executing something completely different
10:56:23 <Shockk> is that the correct understanding? (my kernel is mapped to the same place, but I want to make sure I've got this right)
10:57:46 <bcos_> I don't think anyone has ever written a kernel that's mapped at a different place in each virtual address space
10:58:19 <Shockk> that makes sense, I don't imagine doing so would be a good idea generally
10:58:37 <zid> You'd probably run out of available selectors
10:59:29 <Shockk> so, does mov cr3,eax instantly cause TLB flushes as the assembly code block seems to suggest? i.e. the next instruction would be executed using the new virtual address space?
11:00:38 <zid> Imagine if it didn't?
11:01:18 <Shockk> hmm well it was more I wondered if it only happened on ret or jmp or something like that
11:01:30 <zid> well consider what would happen if it didn't flush
11:01:39 <bcos_> "mov cr3,eax" causes TLB flushes of all pages that aren't marked "global" (unless you're using the PCID feature)
11:01:57 <zid> The TLB would need to manually track if a page was part of the new or old cr3, but then what happens if I reload cr3 like 10 times in a row?
11:02:03 <zid> so it knew which were stale
11:02:11 <zid> gross
11:02:17 <Shockk> hmm lol
11:02:20 <Shockk> right
11:02:24 <bcos_> zid: ..PCID feature
11:03:09 <Shockk> okay this is kind of fascinating to think about, like it's requiring me to change the way I think about program flow
11:03:39 <zid> Just consider the page tables as part of the execution environment and it gets a lot simpler
11:03:53 <gog> program flow is like being a boat on the river except at any moment you could be plucked out of the river by god and put o another one
11:03:54 <bcos_> (PCID = process context ID = put a "process ID" into each TLB entry and ignore the TLB entries that don't have the current ID so that they don't need to be flushed and may still be there next time you want them)
11:04:10 <zid> manually tracked though ^
11:04:11 <Shockk> gog: right, it's more the fact that, er
11:04:15 <bcos_> gog: I want to see a river with loops!
11:04:19 <Shockk> hang on I drew a diagram to try and undersatnd it
11:04:31 <gog> bcos_: canal to a lock to a waterfall
11:04:38 <Shockk> https://i.imgur.com/hLnCj9z.png
11:05:11 <Shockk> as far as I understand, in this example, Task 1 would call cswitch, and then it would appear to return, but would actually be returning from the cswitch that Task *4* called
11:05:17 <Shockk> that's the part that fascinates me
11:05:43 <gog> as far as any particular task is concerned though nothing is amiss
11:05:45 <zid> Shockk: yea that's a silly way to think aout it
11:06:50 <zid> Although I must admit to being a little tickled to think of this as 'the program runs the kernel'
11:07:11 <zid> That is, the kernel never runs, the userspace applications just decide to become the kernel for a little bit sometimes via syscalls
11:07:12 <Shockk> hmm lol
11:07:20 <zid> because the kernel is in their address space, and they're the one calling it..
11:07:22 <Shockk> aside from timer interrupts you mean
11:07:24 <gog> cooperative multitasking?
11:07:48 <gog> as opposed to preemptive multitasking
11:08:01 <zid> Also, I still haven't written my god damn allocator it's been like 2 days
11:08:04 <gog> that makes userspace more complicated tho
11:08:06 <zid> I've been through 10 crap designs
11:08:31 <gog> and a program can simply not cooperate and never cede control
11:09:44 <Shockk> oh, I was thinking of preemptive when I drew that diagram, Task N was meant to be the timer interrupt handler of the kernel while that task is runnin
11:09:45 <Shockk> g
11:10:24 <zid> If you're thinking about timers then just think that the task is entirely suspended
11:10:31 <zid> you just happen to have stolen its kernel mappings
11:11:03 <Shockk> I think I'm about ready to try and actually implement tasks now either way
11:11:08 <Shockk> fun!
11:12:32 <zid> Shockk write my allocator for me
11:12:41 <Shockk> I don't have one yet
11:12:41 <Shockk> :(
11:12:45 <zid> perfect
11:12:48 <Shockk> hahaha
11:12:56 <zid> I already sort of designed it a little, so you just need to write a couple functions for me
11:13:37 <Shockk> but I have to write my multitasking and scheduling and stdin and things
11:13:44 <zid> nah
11:14:19 <bcos_> Shockk: All of those things need to allocate memory
11:14:36 <Shockk> :(
11:15:23 <Shockk> oh, I guess the collection of tasks does need to allocate memory
11:16:36 <bcos_> Typically each process has a "process data block" and each thread has a "thread data block" and they get allocated (and then freed when the thread or process terminates)
11:16:51 <Shockk> ah right
11:17:20 <zid> bcos_: wanna tear my 10th design to shit for me? I already designed a few that sucked balls
11:17:21 <Shockk> the process data block is the space available when allocating memory with malloc, right? and also its stack
11:17:39 <bcos_> zid: Design of what?
11:17:41 <zid> allocator
11:17:59 <bcos_> A file handle allocator?
11:18:06 <zid> physical range
11:18:16 <zid> It's a combination tree of bitmaps + list of lists for various sizes
11:18:28 <zid> It's going to be *gnarly*
11:18:38 <bcos_> For physical memory?
11:18:40 <zid> ye
11:19:02 <zid> I could probably just do 2MB and 4K but I decided to go nuts
11:19:25 <bcos_> Ok
11:20:04 <zid> So the plan is, initially all physical memory is in a linked list in the 2MB slot, and if a smaller request comes in it's broken and re-inserted into the other lists except for the requested page
11:20:19 <zid> and it's all tracked by an additional series of bitmaps to allow free() to merge them back up
11:21:34 <bcos_> That doesn't sound bad
11:21:55 <zid> It does sound pretty gnarly though right
11:23:50 <bcos_> Would need something else for physically contiguous buffers (for device drivers), plus some way to gather missing 4 KiB pages in an "almost unused" 2 MiB page
11:24:29 <zid> No, these are physically contigous
11:24:35 <zid> That's why it's like it is
11:24:57 <zid> all the 2MB entries in the top list are 2MB aligned, precisely for device driver reasons (and also large page mappings)
11:25:16 <zid> they won't get re-inserted into that list until two 1MB pages are returned
11:26:00 <bcos_> So your top list is sorted, in case a device driver wants 4 MiB of physically contiguous RAM?
11:26:12 <zid> No, but I could make a 4MB list if I needed to
11:26:16 <zid> I'd just have to change an enum
11:26:38 <bcos_> So your bottom list is sorted, in case a device driver wants 8 KiB of physically contiguous RAM?
11:26:49 <zid> it doesn't have to be sorted, the bottom list is 4k
11:26:56 <zid> the n-1 list contains 8K pages
11:27:05 <zid> which are just two contiguous 4k pages glued together
11:27:32 <bcos_> D'oh
11:27:56 <zid> each list is just a 'stack of the right size' and they automatically merge and go up a level if possible when returned
11:27:56 <bcos_> I thought you only had 2 lists - one for 2 MiB and the other for 4 KiB
11:28:04 <zid> oh sorry
11:28:44 <bcos_> ..for "many lists with many sizes" it's a mostly inefficient because no other sizes are used often enough
11:29:19 <zid> yea I doubt I will really need contiguous stuff other than 2MB
11:29:28 <bcos_> (it's literally only device drivers, and often only when they're initialised; so you don't want every single allocation to pay for the overhead of that)
11:29:30 <zid> most things can just deal with being behind virtual memory and using a random 4k physical address
11:29:50 <zid> hmm yea if I limit it to initialization time
11:29:57 <zid> I don't need the other lists
11:30:32 <zid> I'd start with a list of 2MB and 0 4k, device drivers would pull some 2MB pages out, then as requests come in all the remaining 2MB pages would be broken into 4kB pages, then I never merge them
11:30:39 <zid> and I end up with 0 2MB pages left
11:31:15 * bcos_ normally splits physical RAM into zones and uses different allocators for different zones (e.g. 0 to 16 MiB = zone managed with a bitmap to make device drivers happy, 16 MiB to 4 GiB = zone managed with free page lists that never has to care about physically contiguous allocations)
11:31:44 <zid> I didn't want to do arbitrary amounts of 2MB pages
11:33:52 <zid> bcos_: Okay so if I lose the 2MB virtual memory pages, I could easily simplify this down to two lists and the 2M never gets refilled you think?
11:35:12 <bcos_> Drivers aren't always started during boot - e.g. user plugs something new into USB slot after OS has been running for 43 days..
11:35:22 <zid> oh right, THAT'S why I wanted that
11:35:25 <zid> I knew there was a reason lol
11:35:38 <zid> keeping 4k fast but keeping 2M available, that seems to be the bastard
11:36:15 <Shockk> is there any way to write this: #define IRQ1 (IRQ0 + 1) so that it's usable in a switch case?
11:36:15 <zid> maybe I should only merge at allocation time for a 2M page then?
11:36:20 <Shockk> or do I have to manually write out the value?
11:36:27 <zid> enum
11:36:30 <zid> they auto-increment
11:36:37 <Shockk> hmm that's a good idea
11:36:43 <zid> plus you get the checking from the compiler
11:36:47 <zid> for missing enum labels etc
11:37:38 <zid> bcos_: So like, still ditch the 8k-1M lists, but 2MB pages are just 'slow' and check the 4k bitmap for a string of 512 pages or something?
11:38:58 <bcos_> When freeing a 4 KIB page, check if all 512 of them become free
11:39:20 <zid> hmm? Merge at free time?
11:39:26 <bcos_> Yes
11:39:31 <zid> it'd give you an index to start from I guess being the benefit there?
11:39:53 <bcos_> ..but that on its own won't prevent all of the 2 MiB pages being split up (e.g. 511 pages in each one free)
11:40:06 <zid> rather than checking the entire physical space during a 2M alloc for a contiguous region
11:40:14 <zid> we just check on return of a 4k page if that 2mB region is now free
11:40:47 <zid> could easily do that with some masks and shifts I guess, and it should just be a single cache line
11:41:17 <zid> if(memcmp(ALIGN_TO_2MB(page), 0xFF, 16) return_2M(ALIGN_TO_2MB(page); or something
11:41:52 <bcos_> Can do it locklessly..
11:42:03 <zid> bah locks
11:42:12 <zid> I'm more than happy to have a BKL honestly
11:44:40 <zid> I'm interested though, how would you do it locklessly?
11:45:04 <zid> cmpxchg only goes up to 8 bytes afaik?
11:45:54 <bcos_> A loop - cmpxchg 32 bits at a time and if you find 32 bits that can't be cleared reverse what you did (set the bits that you cleared)
11:46:56 <zid> how? What if someone steals a page from the beginning and my loop is in the mdidle, then gets to the end and goes "yep, everything was a 1"
11:47:03 <zid> but actually page 3 is now a 0
11:47:16 <zid> oh we're cmpxchging it with 0s?
11:47:23 <zid> my mind is not in the right place for this bear with me :p
11:48:35 <bcos_> With "0=allocatable or mergeable, 1 = not allocatable or mergeable", you'd cmpxchg to set 32 bits from 0 to 1 (and if you fail you'd go back and set them to 0 again)
11:48:59 <zid> right, you just flip them all so that nobody tries to steal a page
11:49:05 * bcos_ nods
11:49:23 <bcos_> Of course would be nicer with cmpxchg16b
11:49:24 <zid> Okay I am liking this strategy more, thanks, I knew asking you would be useful
11:49:30 <bcos_> :-)
11:49:32 <zid> AVX-512 time
11:49:50 <zid> I already threw away a load of shittier designs :p
11:50:04 <bcos_> I don't think AVX-512 gives you an guarantees about atomicity
11:50:20 <zid> lame, it doesn't even have to be coherant, just atomic
11:51:15 <bcos_> You'd still need an "atomically{compare if changed then set if not changed}"