channel logs for 2004 - 2010 are archived at http://tunes.org/~nef/logs/old/ ·· can't be searched
#osdev2 = #osdev @ Libera from 23may2021 to present
#osdev @ OPN/FreeNode from 3apr2001 to 23may2021
all other channels are on OPN/FreeNode from 2004 to present
http://bespin.org/~qz/search/?view=1&c=osdev2&y=23&m=1&d=21
01:28:00 <geist> it's a fairly nice day here today. cold, but right on the usual. clear sky which is odd
01:28:00 <geist> for this time of year in the PNW
01:29:00 <geist> but that's also why it's coldish, usually coldest days are clear
09:41:00 <ddevault> rate my docs https://ares-os.org/docs/helios/capabilities/asid-control/
09:41:00 <bslsk05> ares-os.org: ASID control | Ares
09:42:00 <geist> looks alright
09:44:00 <zid> That's an odd syntax but the styling on the page is nice
09:52:00 <dinkelhacker> FireFly: Here in Stuttgart it's not snowing but its cold^^
10:10:00 <heat> morn
10:27:00 <zid> oh, can we have the cat instead?
10:53:00 <heat> ...and other deeply hurtful things you can say
10:53:00 <heat> ;_;
10:54:00 * mjg snaps theo to give heat a hug
10:55:00 <heat> aww so wholesome
11:00:00 <heat> lea startup_secondary_64(%ebp), %eax
11:00:00 <heat> push %eax
11:00:00 <heat> no way to shorten this is there?
11:01:00 <heat> push startup_secondary_64(%ebp) just loads from the calculated address AFAIK
11:01:00 <Mutabah> not afaik
11:01:00 <Mutabah> why bother? It's not hot code, right?
11:01:00 <heat> yeah
11:02:00 <heat> just seems kind of off
11:04:00 <mjg> premature optimizaiton vibes from heat
11:04:00 <heat> annoying how 32-bit does no sort of pc-relative addressing
11:05:00 <mjg> annoying how register starved it is
11:05:00 <heat> (and how x86_64 pc-rel is explicitly opt-in)
11:05:00 <heat> my phys relocatable code is now peppered with lea sym(%ebp) where ebp is a load bias
11:06:00 <sham1> x86 should have had way more registers than it did. Thankfully AMD64 did fix that
11:06:00 <moon-child> 'annoying how 32-bit x86'
11:06:00 <moon-child> I agree
11:10:00 <heat> early boot x86 seems to have the NUTS to run a lot of C code
11:10:00 <heat> (eg efi stub crap)
11:12:00 <kazinsal> hmm. got two months to crank out my usual planned april fools joke that I never actually get around to doing
11:13:00 <kazinsal> maybe I'll do it this year
11:36:00 <Jari--> should port my Windows CRPG to the OS project I have
12:56:00 <mrvn> heat: what do you need OC relative addressing for when all you can do is run one application at a time?
12:56:00 <mrvn> s/OC/PC/
12:58:00 <mrvn> geist: cloudy, dark, 0°C, one might say dreary.
17:35:00 <zid> gog: https://vxtwitter.com/fuckedupfoods/status/1616594990234214406 Is this how they do on the island?
17:35:00 <bslsk05> twitter: <fuckedupfoods> why https://video.twimg.com/ext_tw_video/1616594804875354115/pu/vid/788x1080/XwiDaaMiIb5bjCOM.mp4?tag=14
18:44:00 <mrvn> zid: You've got to love lava-cake, I mean steak.
19:04:00 <ddevault> highly tempted to put x86 support on my never-implement list
19:04:00 <ddevault> as in 32-bit
19:13:00 <mrvn> you haven't already? What are you thinking?
19:17:00 <ddevault> I support x86_64
19:17:00 <ddevault> my language does not support 32-bit targets yet
19:19:00 <gorgonical> heat: cntpct_el0 ticks at a rate of what?
19:19:00 <gorgonical> Like I have a difference in ticks of cntpct_el0, is that ticks of the general system clock, e.g. frequency?
19:22:00 <heat> gorgonical, i dont know, im the wrong person for spontaneous arm64 questions
19:22:00 <heat> isn't the arm64 timer at a fixed frequency?
19:22:00 <clever> gorgonical: on 32bit arm, the answer is in CNTFRQ, that register has no actual control over the rate, and is purely just somewhere for the bootloader to store the answer, so kernels can later read it
19:22:00 <ddevault> I have some code to program it to 100 Hz cargo culted from toaruos which you can cargo cargo cult cult if you just want a timer of some kind
19:23:00 <gorgonical> ah
19:23:00 <clever> gorgonical: linux will even divide by zero if you dont initialize that register!
19:23:00 <heat> well, no shit
19:23:00 <gorgonical> Oh then that's good, it means it's working. I'm doing some op-tee profiling right now
19:23:00 <heat> it's an architectural reg
19:23:00 <clever> and i do see a CNTFRQ_EL0 in the armv8 docs
19:25:00 <clever> heat: i'm just a bit surprised there was no check that its valid, and it just goes directly to dividing
19:25:00 <clever> but then again, every working bootloader sets it, so its usually not missing
19:25:00 <clever> my fault for creating an invalid bootloader
19:25:00 <heat> i mean, it's an architectural thing
19:26:00 <heat> linux should not try to interpret it
19:26:00 <heat> in fact, "On a Warm reset, this field resets to an architecturally UNKNOWN value."
19:26:00 <clever> > This register is provided so that software can discover the frequency of the system counter.
19:26:00 <heat> so it may not even be 0
19:26:00 <heat> could just as well default to all-1s or dont-care or something
19:26:00 <ddevault> EFI spec says it has to be programmed with the timer frequency on ARM
19:26:00 <ddevault> so it's valid to rely on it at least in the context of EFI
19:27:00 <heat> EFI spec? more like typoefi spec
19:27:00 <heat> haha am ir ight
19:28:00 <mrvn> is anything using warm reset?
19:29:00 <heat> sure, just reboot
19:29:00 <mrvn> heat: that jumps back into the bootloader so that's fine.
19:30:00 <mrvn> What does kexec do? Does it set the reset vector to the loaded kernel and reset?
19:31:00 <heat> i don't think kexec ever resets anything
19:32:00 <mrvn> probably just always jumps straight to the new kernel. But what else would reset AND skip the firmware that sets the CNTFRQ again?
19:32:00 <mrvn> Secondly where else would linux get the frequency from?
19:34:00 <heat> why are you overanalyzing this
19:35:00 <mrvn> beware of timers that change with cpu / vpu frequency scaling.
19:51:00 <geist> by the time the OS is running, CNTFRQ is supposed to already be initialized if the previous layers behaved themselves
19:52:00 <geist> so if it's set i think you can safely rely on it, if it's not then i guess you have to go find the frequenty to shove into it from the device tree, or Just Know
19:52:00 <geist> but other than that the arch timer stuff on armv8 is quite nice. has good guarantees
19:58:00 <heat> its not fun unless you calibrate it yourself using two other timers
20:01:00 <zid> tscpithpet
20:01:00 <heat> lapiccccccc
20:02:00 <heat> acpi too
20:02:00 <heat> you're not lacking any timer choices are you
20:03:00 <heat> /technically/ the rtc too although that's stretching it, not much of a timer although it can have a stable IRQ tick (128Hz IIRC?)
20:05:00 <sham1> "if the previous layers behaved themselves" That's assuming loads
20:27:00 <mrvn> heat: real men have a LED and photodiode at an exactly known distance from each other and measure how long it takes for light to travel. :)
20:34:00 <geist> even if the tick rate sucks, you can still fairly accurately detect the 'edge' of a rtc tick
20:34:00 <geist> fairly though, depends on how fast it is to access the IO port, etc
20:46:00 <heat> yeah sure
20:47:00 <heat> although you're missing the joke that x86 has N timers/clock sources and they all either suck or serve simply to calibrate the TSC :v
20:48:00 <heat> except in older/weird systems where they can actually sometimes serve a purpose! (in practice probably just the hpet right?)
20:48:00 <geist> yeah
20:48:00 <geist> well, i mean the whole 'TSC as a timebase' is relatively new
20:48:00 <geist> like reliably in the last 10 years, with constant/invariant TSC
20:49:00 <geist> otherwise you were always ysing some of the old stuff
20:49:00 <geist> HPET itself is also relatively new, 2000s
20:49:00 <heat> it's very annoying that the only timers/clocksources you want to use do NOT have stable/easily knowable frequencies
20:49:00 <geist> PIT does, so it's always the ultimate fallback
20:49:00 <geist> and/or the thing you calibrate everything against
20:49:00 <heat> all the other ones do. HPET, PIT, RTC, ACPI-pm
20:50:00 <geist> yep, tough HPET is variable in the sense that you have to read it
20:50:00 <geist> PIT (and HPET i think) also have the property that overclocking doesn't fuck it up
20:50:00 <geist> that does affect TSC i think, and is still a hazard
20:50:00 <heat> really?
20:50:00 <heat> so 15h isn't reliable?
20:51:00 <geist> yah, one of the reasons overclockers will say use HPET, etc. and/or reason you may want to manually calibrqte the TSC on boot even if you have the cpuid that says its at this freq
20:51:00 <geist> since that's based on the assumption the base oscillator is running at the particular freq
20:51:00 <geist> but i think it only comes into play if someone overclocks by bumping the base freq, which is much more rare
20:51:00 <heat> I would expect cpuid 15h to be able to understand what the bus frequency is and simply give a new value
20:52:00 <geist> no, i dont think it understands that the osc is moved
20:52:00 <heat> at the end of the day the CPU must be aware of the adjustment
20:52:00 <geist> but i suppose the bios could update that field, though the resolution is whole mhz iirc
20:52:00 <geist> yah and could always figure it out by calibrating TSC against something else
20:53:00 <geist> and no the cpu doesn't need to be aware of it. if the point is you're overclocking the system by bumping the base frequency, you're officially pushing it out of bounds, so the cpu can still assume it's running at 100 * N where you have now set it to 101 or 102 or whatnot
20:53:00 <mrvn> geist: the RTC isn't all that accurate either.
20:53:00 <heat> 15h is also silently misleading on hypervisors as you notice with KVM :v
20:54:00 <heat> geist, I doubt the CPU does not have a hand in setting its base frequency as well
20:54:00 <geist> mrvn: debatable. it's possible at the seconds range, aside from observation error (time to observe an edge on it, etc) it may be PPM pretty good
20:54:00 <geist> heat: sure it doesn't. that's the input oscillator to the whole thing
20:54:00 <mrvn> heat: he is talking about someone desoldering the oscillator and putting in a faster one. The CPU/BIOS only knows about the selected divisor.
20:54:00 <geist> if you have an overclockable board, it may have a VDSO there that the bios can bump out of spec
20:54:00 <geist> exactly
20:55:00 <mrvn> geist: no, the hardware itself often has incredible errors for a clock.
20:55:00 <geist> but i dont think this sort of overclocking is that popular nowadays
20:55:00 <heat> >vdso
20:55:00 <heat> hehe kernel
20:55:00 <heat> geist, anyway so why are certain CPUs not overclockable then?
20:55:00 <geist> because their multipliers are probably fixes
20:56:00 <mrvn> geist: like second per day drift
20:56:00 <geist> mrvn: a) that's a very very very bad RTC, and b) that may still be better than the regular oscillator on the board
20:57:00 <mrvn> geist: Hey, the clock on a C64 drifts 30 minutes per day easily. :)
20:57:00 <geist> my point is RTCs *should* be drifting by no more than a few ms per day, and though that may be bad by atomic clock purposes, that may actually be better than the raw osc on a modern machine
20:57:00 <geist> depends on what their PPM rating is
20:57:00 <geist> but generally RTCs are run by a 32khz crystal with good PPM rating
20:57:00 <mrvn> geist: yeah, *should*. there are cheap companies out there
20:58:00 <geist> yes yes there are always outliers but they may also put an even shittier main oscillator of they're putting in shitty stuff
20:58:00 <geist> my pioint is the RTC is not just universally bad. and in many cases may actually be a *good* thing to calibrate against
20:58:00 <mrvn> Linux has software to correct the drift of the RTC. Does it also include this for calibrating the other timers?
20:59:00 <geist> i'd assume yes. once you have a modern networked machine you just synchronize to something external, and then it's just a software problem
20:59:00 <mrvn> i.e. when I restore the drift correction on boot does it also correct the other timers?
20:59:00 <geist> i'd assume
20:59:00 <heat> i hate time
21:01:00 <mrvn> A drift of 1PPM would be 1000 ticks of a GHz timer. Sounds like that should matter.
21:01:00 <mrvn> heat: there is always too much or too little of it
21:03:00 <geist> though the main oscillator will probably be something like 25 or 100mhz on a modern PC
21:03:00 <geist> re: your C64 that was probably a simple RC oscillator, which are terrible, but very easy to construct
21:03:00 <mrvn> geist: IRQ driven clock
21:03:00 <heat> 1) it's objectively bad and horrible and we'll never figure out true time (atomic clock what?) 2) it's silly 3) it's relative 4) i already had to listen to fuchsia nerds talk way too much about time and now I have to listen to #osdev nerds too
21:04:00 <geist> which means in this context?
21:04:00 <geist> (mrvn that is)
21:04:00 <geist> heat: haha well john, if you knew him, he'll talk to you about time in meatspace too
21:04:00 <heat> the superior timing technique really is the jiffy
21:05:00 <mrvn> geist: It's not a hardware counter. The time you spend incrementing the clock is lost or you drop IRQs or something. can't remeber.
21:05:00 <mrvn> heat: long live the BOGOMIP
21:05:00 <heat> geist, ngl that "you can drop a second over a whole day" kinda scared me
21:06:00 <geist> yah that being said i've only seen one motherboard personally that had bad drift like that, though honestly all modern PCs you typically just clock sync with the network
21:06:00 <geist> so youwouldn't really know if it drifts
21:06:00 <heat> like, surely this means CLOCK_MONOTONIC can also drop a second over a whole day. does this mean long-range timing with CLOCK_MONOTONIC is unreliable?
21:06:00 <heat> what now? cry?
21:07:00 <heat> i was promised my TSC would work for 200 years but now Intel is also lying to me
21:07:00 <mrvn> geist: you can turn of saving the clock to the RTC on shutdown and then keep track of and correct the drift.
21:07:00 <mrvn> heat: monotonic means it doesn't go back in time. Not that it's accurate
21:08:00 <geist> heat: yes, that's correct
21:08:00 <geist> that's why modern software uses external stuff to compute the delta between monotonic nad whatever wall time is
21:09:00 <geist> but in general the kernel really only deals with fairly short term timelines in general, so it's not that big of a deal
21:09:00 <geist> longer term stuff is probably handled in user space anyway
21:09:00 <heat> I had the idea linux never used the RTC except at boot
21:09:00 <mrvn> time only matters when you talk to other computers anyway.
21:09:00 <geist> suspend and whatnot throws a huge spanner in it though
21:10:00 <geist> or if you're doing audio/video stuff, but a pro setup would use external clock syncs anyway
21:10:00 <mrvn> heat: it does
21:10:00 <mrvn> geist: does a PPM error matter for audio/video? Is your hearing that good?
21:11:00 <heat> i was pretty surprised when I found out CLOCK_REALTIME had nothing to do with an RTC
21:11:00 <geist> it matters if you have multiple machines drifting from each other
21:11:00 <mrvn> 22:09 < mrvn> time only matters when you talk to other computers anyway.
21:11:00 <geist> or the sound card is sampling at a rate that drifts from the main cpu oscillator and you start picking up drift
21:11:00 <geist> and then it's the software's problem
21:12:00 <mrvn> geist: so your 1000Hz input turns into 1000.0001Hz. does that matter?
21:12:00 <geist> audio folks will go on ad-nauseum as to why clocks really matter a lot
21:12:00 <geist> eventually, yes.
21:12:00 <geist> eventually as in over the course of maybe minutes, etc
21:12:00 <geist> so it's not an un-real problem
21:14:00 <mrvn> geist: do you make all your microphone cables the same length so you don't get signal delays in there too?
21:14:00 <mrvn> +uneven
21:15:00 <mrvn> I heard they do that to the network cables in datacenters for traders so none of them has an advantage by sitting closer to the switch.
21:15:00 * geist shrugs
21:15:00 <geist> i've also heard that synchronizing audio and video playback when showing a movie/etc on a desktop is actually kinda tricky
21:16:00 <geist> since the audio and video pipelines are usually completely different paths, and at the end of the day the screen/cpu/sound card may have different syncs
21:16:00 <geist> ad humans are pretty good at detecting audio/video drift
21:16:00 <mrvn> horribly difficult. But that's because both the audio and video have unknown and unmeasurable delays
21:16:00 <geist> yah, and even if the delays are known the clocks the hardware is running on are probably different
21:17:00 <mrvn> they are offset but don't drift unless your software screws up.
21:17:00 <geist> sure they drift, if your sound card says it's running at 48khz but it's really 48.001
21:17:00 <geist> then eventually you'll have to resample or sample stuff to synchronize
21:17:00 <mrvn> geist: then you will see less audio samples getting consumed than video frames and adjust the framerate.
21:17:00 <geist> or it's running at 48khz perfectly but your cpu is slightly off, so the systems monotonic is off, etc
21:18:00 <geist> indeed. obviously there are solutions
21:18:00 <geist> but point is its a case where time drift matters
21:18:00 <geist> *without* other computers involved
21:18:00 <mrvn> you also get cases where the audio is 48kHZ but the cheap onboard chip only does 42KHz.
21:19:00 <mrvn> geist: My point was that you don't notice a 48.000 or 48.001 sound card frequency.
21:20:00 <mrvn> The software also has to cope with 29.95Hz vs. 120Hz monitor frequency.
21:20:00 <geist> okay, not sure what we're arguing about now, so will leave it at that
21:22:00 <heat> fun rtc fact
21:22:00 <mrvn> My point was that if the sound card does 48.001kHz you simply go with that playing the 48kHz sound just that bit faster. You don't start resambling and correcting for the bad clock. It's not something people can hear. That is unless you have 2 computers, 2 sound cards and they start to drift away from each other.
21:22:00 <geist> oooh oooh what!
21:22:00 <heat> the efi rtc facilities are so broken in x86 linux won't let you use them
21:22:00 <geist> yay
21:22:00 <heat> ... ok fun here wasn't the right word
21:22:00 <mrvn> heat: what? You can't set the RTC time and wakeup timer?
21:23:00 <heat> no, it's just that EFI has timing facilities built in (that access the underlying CMOS or whatever) and various particular implementations on x86 have been so broken that linux just gave up
21:23:00 <heat> so
21:23:00 <heat> config RTC_DRV_EFI
21:23:00 <heat> tristate "EFI RTC"
21:23:00 <heat> depends on EFI && !X86
21:25:00 <heat> geist, x86_64 C code should be mostly PIC itself right?
21:26:00 <mrvn> heat: much more than x86 but not where it counts
21:26:00 <heat> even with globals?
21:26:00 <geist> much more so than before, but the codegen will definitely still have to opt into using PC relative address calculations
21:26:00 <geist> vs just slamming a globla constant down
21:26:00 <heat> hrm
21:26:00 <geist> so not by default per se, but much less expensive to make it so (than x86-32)
21:26:00 <heat> I don't know how these MANIACS keep running so much C code at the wrong address
21:27:00 <geist> for something like riscv or arm64 it's mostly pic by default because you dont easily have the 'slam down a large constant address' path anyway
21:27:00 <geist> also remember C++ vtables or any sort of jump table like that is almost always un-pic by default
21:28:00 <mrvn> any address-of in the .data section isn't pic.
21:28:00 <mrvn> s/in the .data section/ even
21:28:00 <geist> it can emit a pc relative lea for that
21:29:00 <mrvn> geist: no, an address stored in memory. Not the address of something.
21:29:00 <geist> ah
21:29:00 <mrvn> in case of vtables you have the address of member functions stored in the struct.
21:30:00 <mrvn> lambdas would be another example.
21:31:00 <geist> yeah all basically a subset of the same thing: precomputed address ofs sitting in a variable
21:31:00 <geist> for fuchsia we switched to relative vtables, but that's an ABI changer
21:31:00 <geist> but it helps reduce the amount of fixups in the rodata/text segment
21:31:00 <heat> geist, anyway how is zircon doing runtime relocation anyway? I've noticed you have a wrapper script for that
21:31:00 <mrvn> I wish for a compiler with position relative pointers. Store all pointers as "address - location where you store it"
21:31:00 <heat> linux also does
21:31:00 <geist> heat: for the kernel? yeah that's exactly what it does
21:32:00 <geist> physboot is going to switch to doing a full ELF load of the kernel here failure soon
21:32:00 <geist> most of the machinery is already in place for that
21:32:00 <geist> s/failure/fairly
21:32:00 <geist> haha freudian skip
21:32:00 <heat> i'm afraid I'll need to wrap my kernel in another kernel (yo I heard you like kernels) that knows the relocations and/or can pass them
21:32:00 <geist> slip. damnit, i have a cat rubbing against my arm which is messing me up
21:32:00 <geist> for KASLR?
21:32:00 <heat> yeah
21:33:00 <geist> yah, frankly im dubious of the need for it in a hobby kernel, much less a real one
21:33:00 <mrvn> heat: me too. A full ELF loader in the boot.S that jumps to higher half.
21:33:00 <heat> geist, my kernel is 99% things dubious for a hobby kernel
21:33:00 <heat> like flamegraphs
21:33:00 <geist> 👍
21:34:00 <mrvn> The question though is: Do you write the ELF loader to be 100% position independent or do you add a relocation stub to it?
21:34:00 <geist> on hat note i'm going to go hack some code and close irc
21:34:00 <geist> i love you all to death but i've just burned 2 hours of the weekend on the internet
21:34:00 <heat> hacker
21:34:00 <heat> <3
21:34:00 <heat> mrvn, writing an elf loader to load another elf is silly
21:35:00 <heat> at that point you just use a blob
21:35:00 <mrvn> heat: what else would load an ELF?
21:35:00 <heat> why would you load an ELF?
21:35:00 <mrvn> KASLR
21:35:00 <heat> <heat> at that point you just use a blob
21:37:00 <mrvn> you want to reimplement all the relocation capabilities needed for KASLR in your own blob format?
21:37:00 <heat> processing vmkernel's relocs and making a new wrapper image with a blob is pretty standard
21:37:00 <heat> particularly the "new reloc format" bit. ELF relocations are not trivial and are slow
21:38:00 <heat> particularly if you have stupid amounts of them like you're bound to have for a large kernel
21:38:00 <mrvn> I don't see speed being relevant. It's a one time thing.
21:38:00 <heat> it's boot time and boot complexity
21:39:00 <heat> KISS
21:39:00 <heat> somehow here in #osdev people reach the "what if we added another boot phase" conclusion too easily
21:39:00 <mrvn> My linux takes 50s to boot. I'm not concerned with time spend on relocations. :)
21:39:00 <heat> you can boot linux under 1s
21:40:00 <mrvn> Either way. YOu have a boot.S that then loads a blob or elf or whatever.
21:40:00 <heat> no, it's C
21:40:00 <mrvn> heat: for some reason dhcp takes 35+s in the KVMs to finish.
21:41:00 <heat> because doing KASLR in asm must be some degrading stuff
21:41:00 <mrvn> heat: C wouldn't be PIC. then you need another wrapper that relocates the boot code first.
21:41:00 <heat> C is PIC if you're careful enough
21:41:00 <heat> so be careful :)
21:41:00 <mrvn> verry fragile.
21:41:00 <heat> tis life
21:45:00 <mrvn> last I tried building the page tables would put absolute addresses into the table instead of generating them PC relative.
21:46:00 <heat> i have plenty of page table building code in C, fully PIC
21:47:00 <mrvn> lucky you. mine didn't.
21:51:00 <mrvn> on the other hand I can constexpr the page table.
21:52:00 <netbsduser> limine nicely provides KASLR support
21:52:00 <netbsduser> very convenient
21:53:00 <mrvn> does that support multiboot?
21:56:00 <mrvn> netbsduser: is the stivale boot protocol it uses a clone of multiboot?
21:59:00 <netbsduser> mrvn: it can do multiboot but stivale is new
22:00:00 <mrvn> looks very much multiboot inspired at least
22:00:00 <netbsduser> the newest protocol is just called 'the limine protocol' and is defined for amd64 and aarch64 now: https://github.com/limine-bootloader/limine/blob/trunk/PROTOCOL.md
22:00:00 <bslsk05> github.com: limine/PROTOCOL.md at trunk · limine-bootloader/limine · GitHub
22:01:00 <mrvn> So after implementing stivale, then doing it again as stivale2 they decided they still made a bad job at it and started fresh with 'the limine protocol'?
22:01:00 <netbsduser> it was a no-brainer for me because it provides you with a sane and sensible state on loading (for amd64: you are in long mode, loaded in the higher-half, there is appropriate pagetables setup for this + a direct map of all main memory)
22:04:00 <mrvn> netbsduser: sure. Sounds wonderfull. My worry would be that next month they switch to yet another boot protocol and the existing one gets bitrot.
22:06:00 <netbsduser> limine protocol v.s. stivale2 for amd64 at least appears to be very similar
22:06:00 <mrvn> How does that direct map work for ia32?
22:07:00 <mrvn> netbsduser: huh? limine protocol v.s. stivale2 are completely different.
22:08:00 <mrvn> request/responce vs. tagged structs
22:08:00 <netbsduser> it doesn't, limine protocol has no definition for IA32 (i presume because the benefit of it v.s. multiboot is minimal for IA32 while for amd64 the benefits are clear)
22:09:00 <mrvn> stivale(2) has x86_64, IA32 and aarch64. No ARM support though.
22:09:00 <mrvn> I guess limine is out. No ARM support and I need that too.
22:10:00 <netbsduser> mrvn: the actual tags/requests appear very similar to me
22:10:00 <netbsduser> i might be missing something but it looks like a straightforward mechanical process to adapt stivale2 kernel to limine protocol
22:12:00 <mrvn> netbsduser: unless I'm mistaken in multiboot/stivale2 you set a bitfield and the bootloader gives you the address of a blob with tags. In limine protocol you put a bunch of requests structs into your kernel (or a special section if you like) and the bootloader puts the address of an reply struct into each request it finds.
22:13:00 <mrvn> limine might be actually easier to parse since you don't have to walk through a blob of bytes extrating the tagged structs.
22:14:00 <mrvn> you could write something like: for (auto &req : requests) { if (req.reply) .... }
22:42:00 <energizer> could there be a dynamic language for low-level systems programming? i can't think of one
22:42:00 <mrvn> forth
22:44:00 <mrvn> The amount of low-level system stuff you need for a kernel is miniscule. So you can use pretty much anything that can call a few asm functions,
22:45:00 <energizer> afaict most dynamic languages are garbage collected, which doesn't seem all that suitable for systems programming
22:45:00 <kof123> asm was reflective in a way, von neumann machine. it is the extra layers (hardware and software) that strive to eliminate that
22:45:00 <kof123> wrote that before i saw you explain dynamic
22:46:00 <mrvn> GC is a bit of a problem. But you can do a microkernel with the core in asm / C / C++ and everything else as instances of your dynamic language as separate processes.
22:47:00 <mrvn> or have an incremental GC that runs a bit after every interrupt.
22:48:00 <energizer> i don't mean to define dynamic as 'uses gc' i'm just noticing that they tend to be that way for some reason
22:48:00 <mrvn> kof123: check out https://mirage.io/ if you know ocaml.
22:48:00 <bslsk05> mirage.io: Welcome to MirageOS
22:50:00 <moon-child> energizer: common lisp
22:50:00 <moon-child> cf https://github.com/froggey/mezzano
22:50:00 <bslsk05> froggey/Mezzano - An operating system written in Common Lisp (180 forks/3313 stargazers/MIT)
22:50:00 <mrvn> energizer: Note that a modern GC takes about the same time as a modern malloc.
22:51:00 <sham1> Yeah, this whole "can't use a GC'd language for systems programming" is frankly BS. Even if we discount refcounts being a type of GC and focus on tracing instead
22:52:00 <moon-child> mrvn: tbf
22:52:00 <moon-child> malloc kind of sucks
22:52:00 <moon-child> but yes gc is great
22:52:00 <sham1> It's all about the impl
22:52:00 <mrvn> moon-child: malloc here stands for other memory management techniques.
22:53:00 <mrvn> For kernel work you just have to take a little bit of care the GC doesn't block IRQs for too long at a time.
22:54:00 <mrvn> If you have non-allocating IRQ handlers you can make the GC run with IRQs enabled.
22:54:00 <moon-child> sham1: the interface may constrain what you can implement. For instance, malloc(3) is necessarily prone to pathological fragmentation because it can't compact
22:55:00 <moon-child> mrvn: yeah, absent hard realtime gc I feel the solution is to just avoid allocating in isrs
22:55:00 <sham1> Having thread-local or in this case even core-local heaps would be helpful, because then another CPU going through a GC cycle most likely wouldn't bother the one currently handling an IRQ
22:55:00 <moon-child> though still have to bound pause times to avoid filling up queues. Still
22:56:00 <mrvn> sham1: run one kernel per core with verry little sharing between cores.
22:56:00 <sham1> Eh, that also works
22:56:00 <moon-child> any remotely good gc will have thread-local nursery collections. But I don't think shared-nothing is the solution
22:57:00 <moon-child> as a lot of what's attractive about a gc is making concurrent algorithms work
22:57:00 <moon-child> for those things that you do have to share
22:57:00 <mrvn> then ocamls GC is not even remotely good. It's not multithreaded at all
22:57:00 <moon-child> I mean, ocaml only just recently got multithreading for the mutator, no? Give it time ;P
22:57:00 <moon-child> :P*
22:57:00 <mrvn> moon-child: yes. multicore ocaml is a WIP
22:58:00 <sham1> Even still. It's been, what, a decade or two
22:58:00 <mrvn> haven't checked recently, like last few years.
22:58:00 <sham1> I get that there's complexity there, but still.
22:59:00 <moon-child> python is _still_ completely single-threaded
22:59:00 <mrvn> isn't perl too?
22:59:00 <sham1> Guido cannot into threading
22:59:00 <sham1> Nor Larry
23:00:00 <moon-child> but at least he can yoooooooooooooouuuuuuuniiiiiiicoooooooode
23:00:00 <mrvn> "
23:00:00 <mrvn> Multicore OCaml project has now been merged into OCaml tada. This repository is no longer developed or maintained. Please follow the updates at the OCaml Github repository.
23:01:00 <sham1> Good
23:01:00 <mrvn> Last commit to the repo is Dec. 9th 2022.
23:02:00 <mrvn> So it could be real recent or they just did a few updates to the repo post merge anyway.
23:08:00 <mrvn> My feeling is that a lot of kernel stuff you want to do per core for one reason or another and it's no big hardship to make inter-core communications explicit. So you can run one instance of the language per core and they can use a much simpler GC.
23:11:00 <moon-child> but then you need to use rcu or some such when you communicate
23:11:00 <moon-child> which fucking sucks
23:11:00 <moon-child> and you will screw it up
23:12:00 <moon-child> if you already have a gc, why not use it?
23:12:00 <mrvn> moon-child: ringbuffer work fine or transfering ownership of pages when sending messages.
23:13:00 <mrvn> A multi-core GC has to use atomics and barriers and needs bits for different cores in the metadata and such. Much more complex and potentially a lot slower.
23:14:00 <mrvn> Alone the thread-local nursery is a big jump in complexity.
23:15:00 <moon-child> you need a nursery to not be crap _anyway_
23:15:00 <moon-child> you need atomics and barriers if you want to have more than one core _anyway_. I'm not sure what you mean by 'bits for different cores in the metadata and such'. But that seems like the same sort of complexity as you see in a good concurrent malloc
23:16:00 <mrvn> sure, but that's a different kind. The thread local nursery has to migrate objects when you share them
23:17:00 <moon-child> you are speaking in very absolutist terms
23:17:00 <moon-child> you have to do _something_ when you share something in the nursery and have many threads. You also have to do _something_ when you share something in the nursery and have one thread
23:17:00 <mrvn> Most GCs do mark & sweep in some form and have some bits to color objects. With multiple cores you need per core bits for the color to make sure every core has marked an object.
23:17:00 <moon-child> it's not clear that the former is significantly more complex than the latter
23:18:00 <moon-child> 'With multiple cores you need per core bits for the color to make sure every core has marked an object' wat
23:18:00 <mrvn> Haeh? With a single core there is no sharing.
23:18:00 <moon-child> single threaded generational gc needs a write barrier to cope with old->new pointers
23:19:00 <mrvn> moon-child: yes, for mutables.
23:19:00 <mrvn> if you have them
23:19:00 <moon-child> my friend
23:19:00 <moon-child> when you are writing a kernel
23:19:00 <moon-child> how do you think you are going to get away without mutables?
23:21:00 <mrvn> ignoring hardware registers you can make everything else purely without mutables. It might not be the nicest but it's not required.
23:21:00 <moon-child> if everything is referentially transparent, what does it mean to share something with another core?
23:22:00 <moon-child> mrvn: mmio? Page mapping? Scheduling?
23:22:00 <moon-child> I mean
23:23:00 <moon-child> a kernel's whole job is to manage mutable structures
23:23:00 <mrvn> On a high level sharing then means nothing. But the GC runs below the language and will mutate. Or do you know of a GC that is referentially transparent?
23:24:00 <moon-child> yes, the gc also has to mutate. I was ignoring that for the sake of argument
23:24:00 <mrvn> mmio you don't have under GC controll so you can have code to mutate them. Page tables you can make referentially transparent.
23:25:00 <mrvn> moon-child: that kind of ignores the main point. The GC mutating suddnely turns into mutations from multiple cores and take is what makes it difficult.
23:31:00 <mrvn> anyway, I'm not saying you must run one instance of the language with it's own GC per core. Just that it's not a big deal to do so. And that a single core GC can be written simpler and faster because it can use cheaper memory access.
23:32:00 <mrvn> So no big barrier for writing your OS in python for example.
23:32:00 <mrvn> one python per core and the GIL problem goes away.
23:36:00 <moon-child> ...
23:37:00 <moon-child> I'm not really sure what you mean by 'can use cheaper memrory access'. Nor 'mutations from multiple cores'. Broadly, I think you're assuming a lot about the gc design, whereas the design space is quite large, and it's not at all clear what would be optimal in a given case
23:38:00 <mrvn> moon-child: if the GC mutates something and that needs to be visible on other cores (i,e. outside the per core nursery) then you need atomic access or memory barriers safeguarding it
23:40:00 <mrvn> and any shared object you have to mark as alife from multiple cores so you have to do that carefully and ensure all cores have done so before you free anything.
23:41:00 <mrvn> "it's not at all clear what would be optimal in a given case" which is the extra complexity I mentioned.
23:41:00 <moon-child> I stand by what I said
23:42:00 <mrvn> A multi core GC simply has to consider more design and architecture problems.
23:42:00 <moon-child> I agree with that
23:43:00 <mrvn> sounds like we agree in prinzipal
23:43:00 <moon-child> I disagree with most of the rest of what you said, for the reasons I mentioned
23:44:00 <moon-child> and: in general, adding essential complexity to core infrastructure allows you to remove it from everywhere else, so it is not obviously a loss
23:44:00 <mrvn> didn't say it would be
23:44:00 <mrvn> "it's no big hardship"
23:47:00 <ornx> why do i get a fault when i assign ss to a non-null descriptor in long mode
23:48:00 <ornx> but i don't get a fault when i assign it to a null descriptor
23:48:00 <mrvn> wrong CPL? Not a data segment?
23:48:00 <mrvn> setting ss to a null descriptor should fault on the next interrupt or syscall
23:49:00 <ornx> cpl/rpl is 0, it looks like a data segment ($3 = {limit1 = 0, base1 = 0, base2 = 0, field1 = 10010000, field2 = 0, base3 = 0})
23:49:00 <mrvn> or function call in generall
23:49:00 <ornx> in long mode?
23:49:00 <ornx> i thought it ignored all of the segmentation stuff
23:50:00 <mrvn> not the cpl/rpl and executable bit and such
23:50:00 <ornx> (hence why base/limit are 0 in that descriptor)
23:51:00 <mrvn> if it ignored all of it then why would you have to set them at all?
23:52:00 <mrvn> not sure what bits ss checks but the specs should say
23:52:00 <ornx> it's not technically necessary i suppose but it seems odd to just leave it as whatever the bootloader set it to rather than setting it to a known state
23:52:00 <mrvn> indeed
23:53:00 <mrvn> potentially the bootloader may never have used a stack and never initialized ss.
23:53:00 <mrvn> Check if setting ss to null is OK. I would have expected it to cause a problem on the next use. But I might be wrong.
23:54:00 <mrvn> did you set ds/es/fs/gs?
23:54:00 <ornx> it seems to work fine - my code sets all the segment registers and then calls some functiosn to draw a picture on the framebuffer in c, so it is probably using the stack for this
23:54:00 <ornx> yeah, it sets all of them and return far returns to set cs
23:54:00 <ornx> s/and return/and then/
23:55:00 <ornx> hm actually, i set cs to a code segment, ss to the data segment i pasted earlier, but the rest i set to null
23:55:00 <mrvn> The manual has a section about how to initialize long mode that should tell you in detail how to setup the segment registers.
23:57:00 <ornx> i'm already in long mode actually, it goes EFI -> GRUB -> multiboot2 , and then i'm in EFI AMD64 mode with EFI boot services still enabled
23:57:00 <mrvn> if setting ss to a data segment at boot works but fails later then that sounds like you are not setting it to a data segment later. Maybe mixed up the number? Or corrupted the descriptor?
23:58:00 <moon-child> I suggest telling boot services to go away before doing anything
23:59:00 <moon-child> does it work if you set ds/es/fs/gs to that descriptor? ss is the only one that's screwed up?