Search logs:

channel logs for 2004 - 2010 are archived at http://tunes.org/~nef/logs/old/ ·· can't be searched

#osdev2 = #osdev @ Libera from 23may2021 to present

#osdev @ OPN/FreeNode from 3apr2001 to 23may2021

all other channels are on OPN/FreeNode from 2004 to present


http://bespin.org/~qz/search/?view=1&c=osdev&y=19&m=2&d=11

Monday, 11 February 2019

12:02:21 <mrvn> should go nicely with the Ryzen 5 2400G, Gigabyte MB and 4x16GB DDR4 ram I ordered.
12:06:34 <mrvn> nyc: The nice thing about software defined page tables is that you can make real simple data structures for a simple OS. E.g. a process simply has 1) a text segment, 2) rodata, 3) data + bss + heap growing up, 4) stack growing down.
12:09:41 <mrvn> nyc: so you can check if a fault is something you might have to add to the TLB with one or two cache line accesses.
12:09:55 <nyc> mrvn: They're truly brilliant when your kernel structures are BAT-like and refill with extremely shallow structures with very small space footprints.
12:10:26 <mrvn> and usually they allow for much more flexible TLB entries.
12:10:36 <mrvn> like lots of possible page sizes.
12:11:15 <nyc> It looks like M8 goes from 8KB to 16GB in powers of 8.
12:12:28 <mrvn> oh yeah, 8KB pages are fun. All those software that assume 4K pages...*BOOM*
12:13:11 <nyc> MIPS can have a 1KB granule/whatever and go up to 256MB.
12:13:57 <mrvn> Many systems can do 64k and that seems to be an optimal size for IO.
12:17:08 <Ameisen> mrvn - did you bring enough for the whole class?
12:17:35 <mrvn> Ameisen: yeah, you can all help me pay for it. :)
12:17:41 <Ameisen> MIPS, thankfully, doesn't define a lot of things like page size (or even paging)
12:17:47 <Ameisen> which made making a 'technically compliant' emulator easier.
12:18:11 <geist> yah i'm fairly certain that paging on mips has had multiple different implementations over the years
12:18:41 <mrvn> Ameisen: on the other hand that means the emulator has to put the right bits into the control registers that tell the kernel what paging to use.
12:18:48 <Ameisen> the spec basically says 'this may be implemented'.
12:18:56 <Ameisen> so, I just mark the feature bit as 0, and done.
12:18:57 <Ameisen> :D
12:19:07 <Ameisen> I suspect RISCV would be easier to emulate though - the instruction format seems more straightforward
12:19:16 <geist> back in the heydey of big mips machines i think forward and backwards compatibility of the system mode stuff was not a concern
12:19:17 <Ameisen> MIPS instructions are a little-bit internally-branchy.
12:19:27 <geist> i've heard of some pretty terrible ones, like r8000
12:19:29 <mrvn> Ameisen: branch delay slots, need I say more?
12:19:41 <Ameisen> well, they were annoying to implement, but consistent
12:19:57 <Ameisen> I'm thinking those instructions where certain values of certain operands completely changed the logic of the instruction
12:20:04 <mrvn> what I hate are the exceptions of when you can not use a delay slot.
12:20:16 <nyc> 32-bit MIPS having page sizes from 1KB to 256MB in powers of 4, 36-bit physical address space, and 512MB kernel virtualspace makes it the ideal showcase platform for what I'm planning to implement.
12:21:12 <Ameisen> The POPXX instructions were annoying to implement
12:21:30 <mrvn> Ameisen: luckily gcc automatically moves instructions into the delay slot if allowed. So you don't have to worry about it.
12:21:32 <geist> kinda still wish i had my old SGI Indy, but it was a pain to drag around
12:21:35 <geist> i sold it a few years ago
12:21:47 <Ameisen> POP06. if rs == rt && rs != 0, it's BGEZALC. If rs != rt && rs != 0 && rt != 0, it's BGEUC
12:21:49 <geist> i do have a little mips dev board in the garage though, has some flavor of debian on it
12:21:54 <mrvn> geist: I have one and a monitor but lack the cable.
12:21:55 <Ameisen> otherwise it's an illegal instruction call
12:22:23 <Ameisen> because it compares the values against one another, it also makes it difficult to just split into two instructions in a table lookup
12:22:42 <geist> riscv has a fairly clever set of 'munge this control register' which are kind of a brain teaser
12:22:56 <mrvn> geist: I also have 4 VoCore: https://vocore.io/v2.html
12:23:07 <mrvn> geist: They really are the nicest little toys.
12:23:31 <geist> oh that's rpetty cute
12:23:33 <nyc> My retrocomputing farm went bye bye during some big medical issues I had back in 2008.
12:23:36 <bluezinc> mrvn: cool little gizmos.
12:23:38 <geist> one day i'll fully grok the new numbering convention
12:23:43 <geist> with the mips Nk thing
12:23:47 <Ameisen> mrvn - what's the GPIO latency on those
12:24:39 <mrvn> Runs OpenWRT if you don't want to bare bone it. Ethernet, USB, SD Card, WLAN, uart if you connect it, powered over USB. Just nice.
12:25:02 <mrvn> Ameisen: No idea. Want to software drive a 1GBit card or what?
12:26:03 <Ameisen> mrvn - curious if it can be used to drive things like motors.
12:26:09 <Ameisen> without running into weird timing issues
12:26:34 <mrvn> Ameisen: SPI, I2C and PWM are there.
12:26:52 <Ameisen> one of the issues I'd run into with the cortex Ms is that while they're fast and pretty deterministic, they don't have that much memory, whereas I have a lot of sequential instructions I want to cache up
12:26:57 <mrvn> Ameisen: to drive some stepper motors you probably want to go bare bones. With linux that will have timing issues.
12:27:10 <Ameisen> presumably, I don't have to run linux.
12:27:44 <Ameisen> though there is realtime linux, I'd like to avoid the overhead of linux altogether.
12:28:01 <mrvn> What motors did you want to drive?
12:28:21 <geist> yah i've done real time stuff with LK, and it worked alright
12:28:45 <geist> on a dual cortex-a9. i could basically guarantee that a thread completely dominated the cpu
12:28:56 <geist> via careful ordering of prioroties, etc
12:30:46 <mrvn> I should rewire one of the VoCores to have 2 or 3 rj45. Make it a ethernet snooping device that you can just plug in the middle of a cable.
12:32:03 <mrvn> https://vocore.io/v2u.html for the docked out version.
12:32:17 <nyc> I miss my Cisco 2511.
12:32:54 <bluezinc> mrvn: with HW PWM, you should be able to drive motors from linux.
12:33:34 <bluezinc> ,s/mrvn/Ameisen
12:33:48 <mrvn> bluezinc: sure. Just don't expect them to stop the µs you hit stop.
12:34:33 <bluezinc> fair. but if they're using PWM for control, they're usually not instantaneous.
12:35:29 <nyc> The good thing about actual physical retrocomputing boxen is that they shipped with firmware.
12:36:16 <mrvn> bluezinc: For a fast 3D printer you want to send out ~1Mio pulses/s in a verry controlled way. If linux schedules your process in the middle your circle won't be smooth.
12:36:42 <geist> yah you want a soc with a built in motor controller, which do exist
12:36:51 <geist> then you can offload a bunch of that into the peripheral
12:37:10 <bluezinc> mrvn: fair, I wasn't considering 3d printing.
12:38:07 <bluezinc> for something like that, I'd probably want a multi-core processor, 1 for Linux/networking/control, and 1 for realtime/control work...
12:38:08 <mrvn> It toally depends on what you want to do. If you just want to make a little car that you can drive around from your handy then that VoCore is probably perfect. Add the camera module and write some basic code in python.
12:38:45 <mrvn> One PWM for left wheel, one for right wheel and off you go.
12:38:55 <Ameisen> bluezinc - I don't use PWM for control.
12:39:06 <Ameisen> generally, I run my own ramps
12:40:12 <nyc`> I would have gone for recent models if I could have afforded them.
12:40:27 <Ameisen> What is the proper mechanism by which to update glibc on a linux system...
12:40:32 <Ameisen> Presumably, you have to update _all_ the libraries at once.
12:40:38 <Ameisen> otherwise stuff stops working pretty quick.
12:40:40 <mrvn> apt dist-upgrade
12:40:49 <jmp9> Hello guys
12:40:54 <jmp9> What does this warning mean
12:40:55 <jmp9> ld: warning: dot moved backwards before `.text.__x86.get_pc_thunk.bx'
12:40:58 <Ameisen> All right, what is the mechanism by which the dpkg for libc6 updates it?
12:41:14 <mrvn> Ameisen: no, libs, especially glibc, are pretty backward compatible.
12:41:34 <Ameisen> last time I tried it, lets just say stuff stopped working pretty quickly while I was copying over the new SOs
12:41:40 <Ameisen> nice segfaults in, say, cp
12:41:47 <jmp9> Also
12:41:56 <jmp9> gcc produces invalid binaries sometimes
12:42:00 <jmp9> and everything got crashed
12:42:16 <bluezinc> Ameisen: how did you do it?
12:42:23 <jmp9> for example if(ascii && ascii >= 0x41 && ascii <= 0x5A) this crash
12:42:27 <mrvn> Ameisen: dpkg is pretty resilent there. packages have depends and predepends and dpkg uses verry few libs.
12:42:28 <jmp9> if(ascii >= 0x41 && ascii <= 0x5A)
12:42:28 <geist> jmp9: good afternoon to you too!
12:42:30 <jmp9> and this work fine
12:42:35 <jmp9> Thanks!
12:42:36 <Ameisen> bluezinc - make install.
12:42:44 <Ameisen> which, from what I could tell, just copied over the libs, and then would ldconfig
12:42:46 <jmp9> I've finished paging and doing keyboard routines
12:42:53 <Ameisen> failed about 1/4 of the way through, as cp stopped working
12:43:09 <mrvn> Ameisen: dpkg doesn't use cp.
12:43:22 <bluezinc> Ameisen: if I recall correctly, cp leaves you half-overwritten in the middle.
12:43:26 <Ameisen> Yeah, I don't know what dpkg does to update it.
12:43:30 <Ameisen> that was what I figured happened.
12:43:43 <Ameisen> I had MOST of the libs updated, but they were just enough incompatible that stuff couldn't run, so it couldn't finish updating
12:43:44 <bluezinc> Ameisen: at risk of saying something very stupid, I think mv works a lot better.
12:43:45 <mrvn> Ameisen: it writes the new file and at the end it links it atomically over the old one.
12:43:58 <Ameisen> so it presumably has to be done atomically.
12:44:01 <bluezinc> because that's just repointing the file table.
12:44:02 <mrvn> Ameisen: rename, not copy. So the old file doesn't get replaced half way through.
12:44:02 <Ameisen> which mrvn just said while I was typign
12:44:21 <Ameisen> mrvn - does it atomically update all of them at once, or just one at a time?
12:44:32 <mrvn> Ameisen: one deb at a time
12:45:05 <bluezinc> Ameisen: you should be able to do it one-at-a-time. Because they're reverse-compatible, it's OK as long as you don't end up with one lib that's halfway between the teo.
12:45:10 <bluezinc> /teo/two/
12:45:12 <mrvn> Ameisen: and incompatible libs must have a new SONAME so they don't replace the old lib at all.
12:46:33 * Ameisen is for some reason fighting a build system again
12:46:42 <Ameisen> why is it not looking for includes in /usr/include/x86_64-linux-gnu...
12:46:53 <Ameisen> >:|
12:47:07 <mrvn> Ameisen: dpkg is also quite paranoid with syncs. It ensures everything ends up on disk properly but that makes it slow.
12:47:32 <mrvn> Doing things the right way is slow. If you don't care there is "eatmydata" to speed it up.
12:48:24 <mrvn> I use that in all my chroots. eatmydata uses LD_PRELOAD to override the fsync call to be a NOP.
12:48:37 <knebulae> this just showed up on reddit, definitely some nifty reading to get up to speed on bringing up AARCH64: https://ashw.io/blog/arm64-hypervisor-tutorial/1
12:48:40 <nyc`> My vague plan was to use the alternative architectures from retrocomputing to get some sort of experience and reputation in areas specialized enough to avoid the rather obviously coming mass layoffs of US-based programmers.
12:49:55 <mrvn> knebulae: nice. now fork it for the RPi3 please.
12:50:36 <knebulae> @mrvn: it's not mine.
12:50:47 <mrvn> bookmarked and stared on github
12:50:54 <mrvn> knebulae: that's why I said fork it
12:50:56 <nyc`> I obviously missed that train.
12:51:05 <knebulae> I read port it. lol
12:51:28 <mrvn> knebulae: you have to port the fork too :)
12:51:32 <geist> stick a fork in it?
12:51:49 <knebulae> I haven't even booted the RPi3 I bought yet.
12:52:04 <geist> nyc`: sadly it's all about javascript and other nonsense nowadays
12:52:17 <mrvn> knebulae: haven't booted mine into 64bit yet. Just used one to replace a broken RPi1.
12:52:28 <geist> for low level folk, best way to get some cred is find a sploit in an intel cpu
12:52:42 <knebulae> @mrvn: I thought I'd have more time to play with it, but the weather caused chaos for me last week.
12:53:08 <mrvn> knebulae: I have one sitting one the shelf for over a year waiting for me to get around to it.
12:53:11 <nyc`> geist: I'm not getting anywhere near webdev. That's all offshoreable if not already offshored anyway.
12:53:54 <mrvn> Next weekend I will setup my new PC so that's another 2-3 weekends gone.
12:53:58 <knebulae> @nyc: that example brings up a uart on arm64; maybe beneficial to you?
12:54:21 <nyc`> knebulae: Probably, yes.
12:54:35 <mrvn> knebulae: the hard part about that is figuring out where the UART is on the specific SOC. Once you know the address it's a standard process.
12:54:56 <jmp9> okay
12:55:05 <jmp9> gcc produces strange kernel
12:55:10 <jmp9> that crashes before even get loaded
12:55:15 <jmp9> "not enough memory"
12:55:18 <nyc`> I'm away from my laptop at this very moment, and probably burned out for the day, though.
12:55:22 <mrvn> jmp9: did you use a cross compiler?
12:55:27 <knebulae> right. I'm still trying to wrap my head around the boot process. I haven't looked, but is the os really loaded by the graphics chip in the RPi3?
12:55:28 <jmp9> uhm
12:55:29 <jmp9> i use gcc
12:55:42 <jmp9> x86_64-pc-linux-gnu
12:55:47 <nyc`> knebulae: Oh, and diving headfirst into a firmware quagmire.
12:56:03 <jmp9> with nodefaultlibs
12:56:06 <mrvn> knebulae: the RPi3 has the VC as main CPU. the arm core is just a co processor. And there is a little tool for the VC to boot it up and load a kernel for you.
12:56:19 <mrvn> knebulae: it's verry much inside out compared to every other board.
12:56:54 <jmp9> is there difference between standart gcc and cross compile?
12:56:56 <mrvn> jmp9: there is a reason all the tutorials on the wiki say not to use that
12:57:02 <jmp9> oh
12:57:03 <jmp9> fuck
12:57:21 <jmp9> I didn't noticed that
12:57:26 <knebulae> @mrvn: I understand.
12:57:27 <geist> yes
12:57:31 <mrvn> jmp9: the standard let you screw up the kernel by giving the wrong options. The cross compiler will give an error for most of the obvious mistakes.
12:57:47 <jmp9> Where I can obtain cross compiler for arch linux?
12:57:47 <geist> you can sometimes make a -linux distro compiler do what you want, but only by extremely carefully overriding everything
12:57:50 <knebulae> @nyc: why do you do these things to yourself? :)
12:57:53 <geist> the far better thing to do is get a cross compiler
12:58:05 <geist> jmp9: look on the wiki. there's all sorts of stuff about building your own or downloading one already built
12:58:12 <mrvn> jmp9: the wiki has instructions
12:58:27 <jmp9> ok thanks
12:58:55 <mrvn> jmp9: you also need to write your own linker script
12:59:09 <jmp9> I've done this
12:59:16 <jmp9> I rewrited it for paging
12:59:21 <mrvn> do you boot using multiboot or uefi?
12:59:26 <jmp9> grub multiboot
12:59:34 <jmp9> as you told me
12:59:36 <jmp9> week ago
01:00:01 <nyc`> knebulae: It's the price of non-x86 architectures.
01:00:04 <mrvn> jmp9: did you try out the example kernel from the wiki?
01:00:10 <jmp9> yes
01:00:32 <mrvn> then compare how that is build with how you build your own.
01:00:44 <knebulae> @nyc: yup.
01:00:49 <mrvn> You must have broken something at some place.
01:01:53 <mrvn> n8
01:02:46 <nyc`> knebulae: I suppose the pain for VAX will be finding the last working version of the GNU toolchain for it.
01:03:43 <aalm> nyc, you might want to take a look at what netbsd uses for it
01:04:35 <nyc`> aalm: Even that probably hasn't been tested with any recent versions.
01:07:15 <nyc`> I guess I should write expect scripts to regression test the various ports in qemu.
01:08:59 <jmp9> Okay guys
01:09:04 <jmp9> While cross compiler downloads
01:09:14 <jmp9> I wanna ask more about VM8086 mode
01:09:26 <jmp9> what I need to use BIOS interrupts in protected mode?
01:13:24 <nyc`> I think I have to use a non-qemu emulator for IA64, too, and it's unclear what sort of shape that's in.
01:23:31 <bcos_> nyc`: Itanium (IA64?)?
01:23:34 <nyc> https://trofi.github.io/posts/199-ia64-machine-emulation.html <-- this is not encouraging
01:23:37 <nyc> bcos_ Yes.
01:23:55 <bcos_> That's..
01:24:40 <bcos_> I don't think anyone (any hobbyist OS developer here or elsewhere) has ever tried (or wanted to try) to support Itanium - you'd be a poineer!
01:25:43 <nyc> bcos_: I'm going to be spread thin enough that I can't really drag the toolchain and emulator with me.
01:25:52 <klange> qemu only ever supported ia64 through KVM, and even that's been dropped by now
01:26:06 <klange> (as in KVM on an ia64 host)
01:27:35 <nyc> bcos_: And from the looks of it, it would be a rescue from a decade or more of bitrot project for both of them.
01:31:06 <bcos_> ..and even after that, you'd have no way to tell the difference between "OS works (always)" and "OS is buggy and only works by accident on one VM"
01:32:31 <geist> yah i fiddled with ia64 a while back, butnothing that exists in source control
01:32:48 <geist> i used to have an itanium box i got off ebay, but i got rid of it about 10 years ago
01:33:35 <nyc> bcos_: I'm in that boat for the whole project because I have no access to real hardware. The difference is how dead the simulators/emulators and toolchain are. There are probably more VAX enthusiasts than IA64 enthusiasts keeping the codebases and engineering samples etc. alive.
01:34:12 * bcos_ thinks "purchase price of new small workstation" is one of the things that contributed to the death of Itanium
01:35:24 <geist> but yeah i oubt anyone wanted to actually try to emulate an ia64 for fun
01:35:41 <nyc> bcos_: By that logic basically everything but x86 is inviable.
01:35:44 <Mutabah> geist: For fun? I dunno... sounds like it could be interesting...
01:35:44 <geist> i think the instrruction bundling itself would actually be kind of a nightmare to emulate
01:35:52 <Mutabah> But I doubt it'd be fun to make it accurate :)
01:36:00 <geist> think of every instrruction as having up to 2 branch delay slots
01:36:29 <geist> certainly would be fairly difficult to efficiently JIT into something like x86
01:36:32 <bcos_> nyc: Depends on funding I guess - there's multiple relatively cheap ARMv8 systems (and emulators for it)
01:37:04 <nyc> bcos_: Not workstations, though.
01:37:14 <bcos_> ..but, yes, 80x86 is much easier for finding test systems
01:37:39 <nyc> bcos_: It's literally the only thing available as a cheap workstation.
01:38:10 <geist> yah
01:39:42 <bcos_> nyc: I that case the "effort to support vs. potential users" ratio is also likely to be bad for alternatives (Itanium, PowerPC, Sparc, ..)
01:39:43 <geist> i'm hoping the thunderx2s will become generally available in workstation form
01:40:22 <nyc> geist: It would make me happy.
01:41:45 <nyc> $ qemu-system-sparc64 -nographic -M niagara -kernel ../bld.nmcp/sparc/sys/sparc/nmcp
01:41:45 <nyc> Could not open option rom 'nvram1': No such file or directory
01:41:45 <nyc> qemu-system-sparc64: Unable to load a firmware for -M niagara
01:42:52 <geist> https://b2b.gigabyte.com/ARM-Server/Cavium-ThunderX2 is the maker i think
01:43:01 <geist> the workstation machines we have at work are basically just that in a workstation case
01:43:05 <nyc> I remember seeing it. It looks great to me!
01:43:23 <bcos_> I worry about ARM workstation (then desktop, then laptop) systems becoming popular while using "custom designed, vendor lock-in" crap and ruining everything for people like us
01:43:47 <geist> probably not. ARM is pushing hard to keep all of that standardized
01:43:56 <nyc> bcos_: The first thing that will take off is laptop because of the reduction in power draw.
01:43:58 <geist> ie, to be a arm server/workstation you *must* implement uefi, acpi, etc
01:44:17 <geist> they have a fairly elaborate spec you must follow
01:44:21 <nyc> bcos_: Possibly even tablet as an extension of cellphones.
01:44:28 <geist> SBSC? small base serve config? something like that
01:44:52 <bcos_> Sure; except the "standard" (server base system architecture) involves UEFI, and UEFI involves secure boot, and Microsoft have said "Secure boot on ARM should be vendor-locked"
01:45:02 <nyc> cellphone -> tablet -> laptop -> desktop -> server is the path to the mass market I'm expecting if it happens at all.
01:45:06 <geist> okay, except that it doesn't *have* to
01:45:11 <geist> you just can't boot windows otherwise
01:45:38 <geist> i seriously doubt that MSFT has much clout in this space, frankly
01:45:49 <geist> they are a minor annoyance in the arm world
01:45:49 <nyc> And if it happens at all, it'll be an uphill battle.
01:46:30 <bcos_> ("vendor locked to Microsoft" I mean - it's their way of trying to make sure nobody else can put a different OS on cheap Microsoft tablets, etc)
01:46:34 <nyc> Actually, server may happen before desktop because server space is more heat-conscious.
01:46:35 <geist> i mean i wish them the best of luck, they justdont have the clout to boss everyone around like they do in other spaces
01:46:46 <geist> nyc: right, and has some markup
01:47:04 <geist> basically if you can achieve higher compute density you can get an edge
01:47:14 <bcos_> Erm
01:47:22 <geist> the initial price of the device and whatnot is not as relevant in large setups
01:47:35 <geist> obviously there are tons of other barriers to overcome
01:47:41 <geist> but it's not *crazy*, just hard
01:47:54 <bcos_> The only way ARM will succeed for desktop/laptop (and maybe workstation) is if Microsoft pushes a version of Windows for ARM
01:47:55 <geist> vs a desktop battle, which makes zero sense right now unless you're apple and you control everything end to end
01:48:01 <geist> nah, apple can do it
01:48:16 <geist> but it makes sense for them to do it if they do
01:48:50 <geist> same with chromebooks. they already have pushed arm into that space, because you dont care that it's running ARM
01:48:57 <bcos_> Hrm - yes, Apple too maybe (but they're likely far worse, in a "hardware must be ours" way)
01:49:07 <geist> it just hasn't been a huge winner, since it only makes sens ein the extremely narrow space that is super barrgain bin chromebooks
01:49:14 <nyc> It'll turn into how many cores and threads you can cram on a die, and die sizes are limited by the physics of fabrication, and fabrication processes are limited by heat dissipation and quantum gunk, and so on, so one will basically run out of room for an incredibly bloated instruction decoder and hardware pagetable walker and so on.
01:49:20 <geist> bcos_: yah absolutely
01:49:58 <geist> plus there are some nice architectural differences in arm that hypothetically make it easier to scale than x86
01:50:04 <geist> weak memory model, no TLB shootdowns, etc etc
01:50:12 <geist> how much they matter in the end is debatable
01:50:56 <bcos_> nyc: Amdahl's law implies "many weak CPUs" doesn't scale (except for embarrassingly parallel scenarios, you need something with good single-thread performance)
01:51:28 <geist> yep, that's why i think something like thundex2 and whatever they do for x3 is starting to get 'real'
01:51:45 <geist> the design and performance of those chips is actually getting up there. pretty similar to intel/amd
01:52:22 <bcos_> I'm starting the think something has gone very wrong at Intel
01:52:22 <geist> on paper they're similar design wise, there's really only one known pattern tht works for modern superscalar machines, so it's a matter of pulling it off
01:52:35 <nyc> bcos_: Nobody will want 1024 chickens instead of a big powerful ox, but there are limits to what can be done with hardware.
01:52:47 <geist> i think it's no longer secret sauce how to do tis stuff. now you can graduate college with a pretty good knowledge of how to build modern high performance cpus
01:52:57 <bcos_> Ideally, I'd want 50 chickens and 4 oxen
01:53:20 <geist> whic is also something arm has been innovating on. the whole assymmetrric processing stuff
01:53:32 <geist> in the end it works pretty darn well for non server stuff
01:53:59 <nyc> bcos_: Hardware hitting the wall means that things besides the quantum chemistry being used to fabricate semiconductors won't erase all the CPU and software design distinctions anymore.
01:54:12 <bcos_> I'm not sure many operating systems truly support it (beyond the simple "big:little for power management only")
01:54:17 <nyc> ugh
01:54:32 <geist> bcos_: oh no, it's quite well understood now
01:54:38 <geist> and yes it's prrimarily about power managaement
01:54:46 <nyc> Hardware hitting the wall means good design for software and such will actually matter again.
01:54:56 <geist> to a certain extend we're hitting a great filter of operating systems
01:55:04 <geist> which ones will make it into the bazillion core world
01:55:07 <geist> >32, >64, etc
01:55:16 <geist> since it's clearly here to stay
01:55:29 <geist> over the next 5-10-20 years
01:55:46 <geist> its' clear here's not some mega paradigm shift on the horizon except to keep doing what we're doing now
01:55:47 <nyc> Fujitsu is shipping boxen with 3072 threads (I forget how many cores).
01:55:52 <geist> exactly
01:56:01 <nyc> 2**16 cores is rapidly en route.
01:56:56 <geist> i have a 256 thread machine under my desk, there are intel servers bigger than that, cell phones have 12 or 16 symmetric cores, etc
01:57:00 <nyc> A bitmap of all the CPU's is getting to the point it will blow the stack.
01:57:15 <geist> rright, i have some work in zircon in the near future to get us beyond 32 and 64
01:57:20 <geist> because of precisely that
01:57:24 <nyc> Intel is shipping 56 threads per package/socket.
01:57:43 <geist> yah i think that's how you get the big one. 56 * 8 gets you 488 i think
01:58:14 <Griwes> geist, out of pure curiosity, what's the limiting factor for your current implementation?
01:58:27 <geist> AMD really upped the 4d3d3d3 for consumers lately too
01:58:29 <geist> so that's lovely
01:58:39 <geist> Griwes: well because the mega big thread lock, which has to be broken apart
01:58:52 <nyc`> unsigned long CPU bitmap.
01:59:02 <bcos_> geist: Yeah, I want it for more than power managent, like... "2 tasks of equal priority, one chicken, one oxen" to actually work in a way that both tasks get equal work done (and not equal CPU time)
01:59:03 <geist> at the moment there are a few statically sized things based on SMP_MAX_CPUs so i have it set to 32 until i can clear that out
01:59:09 <geist> and then yes, unsigned int bitmap
01:59:22 <geist> can take it to 64 easily, but i need to write a class for it to scale it beyond. not a biggie
01:59:51 <geist> bcos_: yeah we're tossing around some ideas for the next gen scheduler with somehting like that
02:00:12 <geist> ie, load the cores based on their capability to try to keep the queues balanced evenly, taking capability into account
02:00:17 <Griwes> I've recently learned of some parts of our driver that take a *global lock* over all devices in the system for some silly operations
02:00:39 <knebulae> @geist: this is what I was alluding to the other day.
02:00:39 <Griwes> I don't understand how things like that pass review
02:00:50 <geist> yah
02:01:25 <geist> eieio has been doing a lot of thinking about the scheduler and is working on a new implementation
02:01:38 <knebulae> @geist: manycore. I actually think I have it, but I can't show until I'm 100% sure.
02:01:40 <bcos_> Hrm
02:01:41 <geist> that along with proper topology we have some pretty good ideas to get it to the next level
02:02:04 <nyc`> Per-something locks aren't enough to fix it all.
02:02:04 <klange> sounds like fun
02:02:30 <nyc`> A lot of times the workload will just hammer on that something.
02:02:35 <geist> yah we haven't yet gone down the lock free data structure route
02:02:36 <bcos_> geist: For 80x86 I was (am) planning to use "(non-bogus) instructions retired" performance monitoring counters as the basis for scheduling (and not time)
02:03:08 <geist> question there is if using those counters is practical on intel and amd
02:03:23 <geist> as in can you just keep them on all the time or does it have serrious power ramifications or whatnot
02:03:43 <bcos_> It's "possible but annoying (due to being model specific)" for all 64-bit CPUs
02:03:51 * geist nods
02:04:47 <nyc> Hubertus Franke had a project like that involving Kalman filters.
02:05:09 <bcos_> With all the power management (CPU speeds fluctuating on a whim due to turbo-boost, etc) it's the "least worst" option I could think of.. ;-)
02:05:37 <geist> well, like many things you can't always get what you want. most of these scheduling algorithms are approximations of the real thing
02:05:41 <nyc> He basically was trying to find threads and/or processes that were using different processor resources e.g. memory vs. integer vs. float to put on the same hyperthreaded core.
02:05:48 <geist> ie, the Perfect Scheduler where you can see infinitely into the future
02:06:03 <geist> the kwizatch haderach of schedulers
02:06:28 <nyc> I think that's what he used the Kalman filters for.
02:06:47 <bluezinc> wasn't expecting to see Kalman filters come up here, but that's cool.
02:09:18 <nyc> I get the feeling it was a brute force prediction method with feedback and the real thought was put into an analogue of the Osterhout matrix for the functional units on the cores.
02:09:28 <klys> geist, would you think systemwide paging becomes a bottleneck with more cores?
02:10:03 <jmp9> do i need disable red zone for libgcc
02:10:06 <jmp9> if my os is 32 bit
02:10:11 <geist> no
02:10:17 <geist> red zone only applies to x86-64
02:10:18 <jmp9> awesome
02:10:35 <geist> klys: potentially. everything is a bottleneck at some point
02:10:40 <geist> it's all just relative to something else
02:10:42 <jmp9> also i get nose bleed
02:10:46 <jmp9> and my laptop all in blood
02:11:03 <klange> hearing some of this stuff makes me feel like I'm a kid playing with lego bricks compared to what's happening in the "real world" of OSes...
02:11:25 <klys> geist, I'm jst trying to think of the most structurated and therefor most difficult to remove bottleneck you might encounter with extra cores.
02:11:28 <geist> nah, i have but one eye
02:11:58 <geist> i think most of the people hat *really* understand this shit aren't sitting around blabbing on irc
02:12:19 <geist> i know i'm certainly always wondering what the hell i'm talking about
02:12:40 <klys> what is truth.
02:12:42 <nyc> Don't ask me. I haven't been on the rocks for 10 years because I'm worth anything.
02:12:43 <geist> i'm just pretty good at knowing what i dont know, and then talking about it
02:12:55 <geist> which makes you seem like you know what you're talking about :)
02:13:39 <nyc> qemu's sun4u seems to not need external firmware images.
02:13:51 <geist> nyc: yeah i dunno where you got the idea that you need firmware
02:13:57 <geist> -kernel usually lets you side step that
02:14:10 <geist> it doesn't mean you boot the same way a real machine would, but it is sufficient for getting your stuff going now
02:14:15 <nyc> geist: A lot of the machine models demand it even with -kernel, e.g. Niagara.
02:14:26 <geist> ah. hmm, maybe it's like alpha's PALcode or something
02:14:59 <aalm> klange, fwiw. you can get "toy implementation" example of rcu locks from wikipedia for your real world "toy OS":]
02:15:07 <nyc> I think a few of the MIPS machines need it, too.
02:15:20 <nyc> aalm: RCU isn't the only lockfree algorithm out there.
02:15:48 <klys> there might be a way perhaps with your pml4 to divide memory among sets of cores, and that would require spinlocks to bridge and fork from an opposing memory tent. except, it would keep logically separate computing going, provided you could also divide the vfs layer along similar bounds.
02:17:41 <nyc> aalm: An old friend of mine used to be really into epoch reclamation (a similar idea to RCU with a different, more explicit way to garbage collect memory that might still be in use by accessors).
02:19:03 <knebulae> well of course, the majority of people that really know this stuff are probably not on irc at all times, because who wants to do their day job all the time for fun?
02:19:26 <klys> supposing you have 256 cores, assign 64 cores to each of four `tents' and memory would run separate system-layer os software. then you could perform logically separate tasks with processes loaded from a read-only vfs and output to a cornered part of VFS.
02:19:29 <nyc> aalm: Very basic stacks and queues and such get very difficult very quickly when doing them completely lockfree without the sort of GC-like policy that RCU and epoch reclamation do.
02:19:42 <nyc> klys: Multikernels have been done.
02:19:53 <bluezinc> klys: it's kernels all the way down.
02:20:02 <nyc> klys: https://en.wikipedia.org/wiki/Multikernel
02:20:12 <knebulae> I keep coming up with kernel on kernel, but the acronym sucks.
02:20:26 <geist> yo dawg i heard you like kernels
02:20:37 <klys> yeah
02:20:41 <knebulae> maybe kernel in kernel?
02:20:44 <knebulae> KiK?
02:20:51 <bluezinc> knebulae: Simple NEsted Kernel?
02:20:53 <bluezinc> snek?
02:21:03 * geist turns into a sandworm and sends everyone down the Golden Path
02:21:30 <knebulae> @bluezinc: more like isolated
02:21:31 <nyc> geist: I'll chip in a few canisters of popcorn.
02:21:33 * klys pulls out his crowbar
02:21:51 <knebulae> @bluezinc: it's a policy thing
02:22:40 <klys> multikernels have message-passing though there;s no shared memory (not even read-only memory images?)
02:22:46 <nyc> http://popcornlinux.org/ <--- well, apparently the only people google is turning up who've had that idea before are a mere Linux distro
02:23:10 <bluezinc> I could go for snek as an acronym...
02:23:23 <knebulae> don't tread on snek
02:23:28 <nyc> Maybe it's more than a distro? https://www.semanticscholar.org/paper/Popcorn-%3A-a-replicated-kernel-OS-based-on-Linux-Barbalace/29d2ae0e8094492882e58f028bfe19abefb304ba
02:23:34 <geist> snek eet birb
02:26:01 <bluezinc> I think I finally found a name for my kernel.
02:26:48 <nyc> I'll be fine with NadiaOS or whatever.
02:28:13 <jmp9> nadia
02:28:14 <jmp9> hah
02:28:17 <jmp9> it russian female name
02:28:28 <klys> geist, do you see an advantage to message-passing over borrowing external (wide) address space for loading read-only vfs mounts
02:28:30 <jmp9> its
02:28:30 <nyc> jmp9: /whois nyc
02:28:50 <jmp9> ehehehehheh
02:29:00 <jmp9> wtf
02:29:03 <jmp9> Nadia?!
02:29:14 <nyc> jmp9: These are my initials.
02:29:15 <geist> nadios
02:29:23 <jmp9> ah
02:30:07 <geist> or SquirrelAndMooseos
02:30:36 <nyc> Nick: nyc
02:30:36 <nyc> Real name: Nadia Yvette Chambers
02:31:00 <jmp9> [adriane@fukurokouji build-gcc]$ $HOME/opt/cross/bin/$TARGET-gcc --version i686-elf-gcc (GCC) 7.2.0 Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
02:31:59 <geist> grats
02:32:03 <geist> i think
02:32:31 <nyc> jmp9: You need a cross compiler?
02:33:07 <geist> they're doing cross development for the kernel, so yes
02:33:15 <jmp9> I've compiled cross compiler from sources
02:33:30 <jmp9> i'm testing it right now
02:36:26 <nyc> qemu docs are a wee bit imprecise about what architecture revisions their sun4u emulation is targeting.
02:37:03 <jmp9> Holy shit
02:37:07 <jmp9> I compiled it with cross compiler
02:37:09 <jmp9> and it works
02:37:37 <jmp9> okay my keyboard works fine
02:38:00 <nyc> jmp9: yay!
02:39:45 <knebulae> @jmp9: \o/
02:39:52 <jmp9> just look at this
02:39:52 <jmp9> https://pp.userapi.com/c849536/v849536897/120de0/j0_qhhPkuno.jpg
02:41:17 <aalm> looks broken, something wrong with your upper case characters?
02:41:28 <jmp9> uhm
02:41:35 <jmp9> i forgot to press shift xD
02:43:04 <jmp9> https://i.imgur.com/FOOaTzK.png
02:43:08 <jmp9> upper case working too
02:43:12 <nyc> There's T1 firmware *somewhere* but it's not obvious where.
02:43:40 <aalm> jmp9, YEEEAHH!!
02:44:00 <nyc> http://download.oracle.com/technetwork/systems/opensparc/OpenSPARCT1_1_6_os_boot.ace <---- that doesn't look like what qemu is complaining about.
02:45:07 <nyc> Could not open option rom 'nvram1': No such file or directory <--- that's a different file.
02:53:07 <nyc> mc->default_cpu_type = SPARC_CPU_TYPE_NAME("TI-UltraSparc-IIi");
02:53:10 <nyc> Got it.
02:55:14 <nyc> I guess that's still v9.
03:04:24 <nyc> Oh boy, I have to go through PCI to get at the serial on a sun4u.
03:11:40 <geist> nyc: eww ick
03:11:49 <geist> in practice it's probably hard coded, effectively
03:11:56 <geist> you can probably just start blatting it out
03:12:37 <nyc> geist: There's a wee bit of obfuscation of the precise address to start bitbanging.
03:12:52 <geist> see the info qtree thing i was talking about before
03:12:55 <geist> sometimes that'll help you
03:13:04 <geist> depending on how much they broke it out as a separate device
03:13:30 <nyc> A combination of info qtree / info mtree and reading the qemu source are my current guidance.
03:36:25 <nyc`> I've gone upstairs for the night. I know where to start poking around in gdb if I can't tell by reading the code before I get the chance (unlikely, since if I could get at the code, I could gdb qemu for it all).
03:46:16 <nyc`> I have a vague idea that 2**16 CPU's would truly abhor cacheline bounces. I think refcounting as is common leads to a lot of that and is often criticized in GC land for its CPU overhead.
03:51:28 <nyc`> The alternative is unclear, though.
04:02:43 <nyc`> Space overhead is high for Baker's treadmill and I'm pig ignorant of concurrent/parallel GC algorithms. I know stop and copy sure isn't it.
04:07:09 <nyc`> I'll have to file that away for later. I'm rather far from approaching that sort of issue.
04:16:31 <froggey> Azul's C4 Java GC was running on 800+ core machines with 100+GB of memory a decade ago. they've published some papers & talks on it
04:17:04 <froggey> afaik Shenandoah (OpenJDK) & ZGC (Oracle) have similar targets
04:23:39 <nyc`> froggey: Very interesting!
04:24:19 <nyc`> froggey: I'll have to go digging for it.
04:25:25 <bcos_> nyc`: Just read a few paragraphs of this: https://news.ycombinator.com/item?id=8541447
04:33:40 <nyc`> bcos_: I'm surprised it's a copying collector.
04:35:42 * bcos_ shrugs - I'm surprised anyone willing to spend the $$$ on an 800+ core machine wants to run Java
04:37:12 <bcos_> (I mean really, how bad does your web page's "java applet" need to be before ... ;-)
04:38:57 <nyc`> Autoparallelizing Fortran compilers usually generate native code, not JVM bytecode.
04:42:54 <nyc`> I guess people are trying to run code that actually pounds the kernel instead of the FPU these days.
04:43:28 <geist> yah azul had some neat machines
04:43:47 <geist> i interviewed there once. they had an 896 core box, was their own risc architecture and their own OS
04:44:06 <geist> later on they predictably switched to building x86 machines and selling the jvm
04:44:13 <geist> but early on they had a whole harrdware solution
04:45:09 <nyc`> I weep for our lost hardware diversity.
04:49:42 <mawk> CLI does only clear the flag in %rflags ?
04:50:04 <mawk> I mean, does RFLAGS.IF is the only source that's checked to see if interrupts are enabled ? isn't there another flag in the LAPIC or elsewhere ?
04:50:20 <geist> that's it
04:50:30 <geist> any other flags at interrupt controllers are masking interrupts at another level
04:50:49 <mawk> nice
04:51:11 <mawk> so when linux says "interrupts must be disabled on entry to protected mode code" I just have to clear RFLAGS.IF
04:51:19 <mawk> and kvm won't send interrupts or anything
04:51:36 <geist> well. probably. i dont know the situation you're talking about
04:51:51 <geist> if you're running as a guest the hypervisor can still trap interrupts without your knowledge
04:51:58 <mawk> I'm the hypervisor
04:52:18 * geist nods
04:52:38 <geist> note that cli doesn't mask NMIs nor does it mask internal exceptions (page faults, #gp, etc)
04:52:46 <mawk> yeah
04:52:54 <geist> think of it as a gate on the external interrupt line that's coming in from an interrupt controller
04:53:01 <mawk> but these won't arrive until linux installs its own handlers
04:53:08 <mawk> if linux code is correct
05:53:31 <mawk> what are the I/O ports that start at 0xf0000 ?
05:53:34 <mawk> uh
05:53:38 <mawk> that's not an I/O port
05:55:05 <mawk> ok so linux is trying to read from 0x413, 0x40E
05:55:15 <mawk> that's bios stuff, let's not give him anything so he won't bother
05:55:53 <mawk> then it's trying to read from 0xF0000 which is the motherboard bios ROM according to osdev
05:55:58 <mawk> why would linux read from the ROM ??
05:56:15 <Mutabah> Looking for the ACPI root?
05:56:30 <mawk> ah
05:56:50 <geist> yep
05:56:51 <mawk> I see
05:56:55 <mawk> but I said acpi=off on the command line
05:56:59 <mawk> maybe it tries to find stuff anyway
05:57:08 <geist> there are other stuff in there, maybe it's falling back to the legacy MP table
05:57:10 <geist> which is also in there
05:57:17 <mawk> ah
05:57:20 <geist> also i think smbios is in there? something like that
05:57:29 <mawk> I see
05:57:40 <mawk> well if it doesn't find any magic maybe he'll just go on
05:58:02 <geist> yah, if you provide something full of zeros it'll probably give up
05:58:11 <geist> and presumably continue to boot with one cpu
05:58:13 <bcos_> Are you sure "ACPI=off" means no ACPI at all (and not just no AML)?
05:58:28 <geist> well it should be able to handle not having acpi anyway
05:58:39 <bcos_> ..like, no multi-CPU
05:58:46 <geist> right
05:58:54 <geist> well, it'll searrch for mp tables and fall back to that
05:59:04 <geist> but if it can't find that it'll be no smp
05:59:08 <mawk> and if mp tables are not there ?
05:59:09 <mawk> ah
05:59:39 <geist> unless there's a way on the command line to just pass in the cpu apic addresses or something, it'll eventually need to find one of the tables
05:59:54 <mawk> isn't the address in a MSR ?
06:00:04 <geist> that's for the current cpu
06:00:09 <mawk> ah
06:00:10 <geist> to find the other ones to boot you need a table
06:00:17 <mawk> right
06:00:29 <mawk> well I'm starting with a single vCPU for now
06:00:37 <geist> and i dont think it was ever safe to try to probe for additional cpus by just blindly starting incrementing APIC ids
06:00:43 <geist> yah prropbably not a biggie anyway
06:00:44 <mawk> since the deadline is 7 hours ago anyway
06:01:05 <geist> plus you can emulate legacy PIC/PIT and probably get by
06:01:13 <geist> prior to imeplementing smp
06:01:21 <mawk> I have an APIC
06:01:24 <geist> well, or that
06:01:52 <mawk> now I just need a serial port, and I'll have something usable
06:02:19 <mawk> letting linux read from it is more difficult, I have to find how to properly send IRQs and acknowledges the EOI
06:02:25 <geist> yah
06:02:28 <mawk> I'm glad linux can boot in such a broken environment
06:02:34 <geist> yah
06:02:40 <mawk> but viewing the docs and the code it's been engineered that way
06:03:07 <mawk> like sentinel values to detect stupid bootloaders
06:03:18 <mawk> then mask out sensitive values to prevent crashes
06:03:33 <geist> yah the linux zero page is a complete disaster
06:03:40 <geist> layers and layers of hacks to keep extending it
06:03:43 <mawk> lol
06:04:16 <geist> and it all goes back to being able to blat the linux kernel directly to a floppy disk
06:04:34 <geist> and having all the boot params be stored somewhere in there in a fairly unstructured way
06:04:34 <mawk> yeah
06:04:37 <mawk> and even from MS-DOS
06:04:41 <mawk> with loadlin.ex
06:04:42 <mawk> e
06:04:51 <mawk> the docs call that a "very hostile environment"
06:05:03 <geist> i had mostly forgotten about that detail until playing with slackware 2 on the 386 i have here
06:05:16 <geist> forgot that you can modify the root= and whatnot with a utility on the floppy
06:06:42 <bcos_> "Loading kernel, please insert floppy disk #5.." ;-)
06:07:02 <mawk> when I first tried linux it was already cds
06:07:45 <mawk> so I never had the pleasure of using floppies that much
06:08:03 <mawk> I remember ordering a free ubuntu cd by mail to try it on my computer
06:08:05 <mawk> now they sell them
06:10:03 <klange> I remember the first time I had code ship in an Ubuntu release, 8.04.
06:23:44 <graphitemaster> I remember installing Windows 95 from floppies, 13 of those DMF ones
06:24:31 <graphitemaster> DMF wasn't some copy protection thing either like people claimed it was but rather just a better floppy that held more than 1.44MB, the fact copying them was hard was because nothing understood how
06:25:13 <graphitemaster> https://en.wikipedia.org/wiki/Distribution_Media_Format
06:26:30 <graphitemaster> Monkey Island came on 10 floppies
06:28:49 <graphitemaster> I should note I didn't do this during the actual time period this stuff came out but like on old hardware that one could pick from the side of the road for free on their way back from school
06:36:44 <mawk> maybe if I send a #GP linux won't interpret the zeroes it reads ?
06:37:06 <mawk> if it's indeed a #GP I need to send for an address that doesn't exist
06:40:05 <mawk> no, I gave it a segment saying it could read
06:40:10 <geist> i think my first shipping code was a bugfix or two in a fs driver in beos r4.5
06:40:15 <mawk> I can't send an exception for no reason
06:42:37 <mawk> I did not ship anything yet, but I have something almost ready for the RTEMS rtos
06:42:43 <mawk> I added a 802.15.4/6LoWPAN stack
06:43:09 <mawk> which is compressed ipv6 for the very small MTU transport that is 802.15.4
06:47:41 <geist> neat
06:49:15 <mawk> incidentally since the linux implementation for the tranceiver I'm using is broken I'll fix it too
06:49:53 <mawk> there's a race condition or two that makes defragmentation fails
08:18:58 <nyc`> Ah, Linux is proof incarnate that retrofitting has limits.
08:19:58 <nyc`> I found that out the hard way when trying to change the semantics of PAGE_SIZE.
08:23:15 <nyc`> Swapping over NFS for Linux was reputedly very invasive, too, though I think that was Peter Zijlstra.
08:28:41 <nyc`> I made an attempt to remove mem_map[] altogether at one point that failed. The most that could be done was to remove a few fields of struct page and make it a per-node/per-zone structure, and ->virtual got resuscitated as well as the global array.
08:30:32 <nyc`> Different design ideologies clashed.
08:31:27 <mawk> I have a kernel panic !
08:31:28 <mawk> on my emulator
08:31:31 <mawk> which means a good thing
08:31:40 <mawk> attempting to kill init blah blah
08:31:45 <nyc`> mawk: =)
08:31:59 <mawk> now I just have to make a good serial port
08:32:04 <mawk> because for now the baud rate is like 2
08:32:44 <mawk> https://asciinema.org/a/96Evs181wElVs68L2PsUBcyJ3
08:35:10 <nyc`> I'm not even trying to do interrupt-driven IO yet. I'm trying to get through arches / CPU types rapid fire first.
08:37:17 <nyc`> I made some attempts to make pagetables a more private data structure, too, to no avail.
08:37:36 <mawk> my stepmother is named zijlstra too
08:37:40 <mawk> maybe they're related
08:38:16 <nyc`> mawk: They're probably both of Dutch ancestry.
08:38:25 <mawk> she's dutch
08:38:27 <mawk> yeah
08:39:30 <nyc`> The Netherlands would be nice. It's a language to do on Duolingo.
08:39:45 <mawk> I need to learn it
08:39:50 <mawk> that's where I'll live in a few months
08:40:32 <nyc`> mawk: :O congratulaciones
08:42:21 <nyc`> mawk: osdev to showcase talent and/or be a portfolio of recent work is my desperate attempt at an escape plan.
08:43:04 <mawk> it did get me an internship for a totally unrelated thing
08:43:15 <mawk> by just saying "I did an attempt at a x86_64 kernel"
08:43:31 <zhiayang> interesting
08:43:35 <zhiayang> i should try that tactic some time
08:43:47 <nyc`> mawk: The theory is that if I do something impressive enough, I might attract interest from visa sponsors in the EU.
08:45:04 <nyc`> zhiayang: You sound like you're far enough along that you might be able to already.
08:45:23 <zhiayang> right, but currently i don't think i have an opportunity to "show off", as it were
08:46:47 <nyc`> zhiayang: I think GitHub and I have no idea where to post announcements to might do.
08:46:52 <mawk> in what country are you nyc` ?
08:47:09 <mawk> compared to USA, EU visas should be easy to obtain no ?
08:47:17 <mawk> you can even buy them if you're rich enough
08:47:26 <mawk> european passports
08:47:37 <nyc`> mawk: I'm in the US.
08:47:51 <geist> bawk!
08:49:45 <nyc`> mawk: Visas are tough all over these days. I'm hoping beyond hope Gaisler or somewhere might be interested in SPARC work or some such.
08:50:41 <nyc`> mawk: I'm even working on Swedish on Duolingo.
08:51:37 <nyc`> (Gaisler does the LEON )
08:55:57 <geist> thought that was for ESA or something
08:56:02 <geist> may have contracted it
08:57:54 <nyc`> Yeah, they do radiation resistant sparc32.
09:01:26 <nyc`> I also had a stint as the Linux 32-bit SPARC maintainer, but it's unclear if that made all that good an impression because of the illness etc. issues around it.
09:02:29 <nyc`> I figure I can signal that I'm serious about legacy with a VAX port.
09:06:09 <nyc`> Whatever it is I do, just make sure that half of Bangalore and Lahore and a good chunk of Shenzhen isn't doing it.
09:11:52 <geist> i think you're going to find it hard to find anyone that even remembers what a VAX is
09:18:39 <nyc> geist: Nah, there are still DoD installations etc.
09:19:12 <nyc> I don't have clearance for those, but they're not the only legacy users.
09:20:05 <nyc> I remember some of the retrocomputing people I knew cashed in big time when 9/11 destroyed some of the Pentagon's old legacy systems, I think PDP-11 and VAX.
09:21:56 <nyc> Conspiracy or no, military people went shopping for legacy systems in the wake of the fires, airplane crashes, etc. in the Pentagon.
09:28:42 <nyc> They weren't rich for life but it was a good chunk of money. They were still on IRC afterward.
09:29:12 <nyc> I'm trying to remember their nicks on EfNet.
09:34:48 <nyc> https://www.computerworld.com/article/2483690/it-careers/india-to-overtake-u-s--on-number-of-developers-by-2017.html <--- part of the osdev strategy is to tie things more to physical access (though obviously not entirely) to combat the whole offshoring trend.
09:36:02 <geist> you do realize that was over 15 years ago
09:36:26 <geist> whatever old vaxen and pdp-11s were still in service then, i'm fairl certain they're pretty decimated
09:37:57 <geist> just sayin, if you're trying to get some sort of street cred for the purpose of getting a job, not sure that's a particularly fruitful tree to bark up
09:40:14 <nyc> Almost 20 years ago now, true.
10:04:47 <nyc> Big groan. The sun4u OpenBIOS is apparently ignoring the kernel I preloaded with -kernel.
10:14:12 <nyc> Latest commit 5c9eb9b on Apr 26, 2006 <--- up-to-date documentation is always encouraging
10:25:07 <mobile_c_> [09:53] <nyc> mobile_c: Most of the time p_paddr and p_vaddr are the same. It only makes a difference for kernels and such.
10:25:16 <mobile_c_> so wtf is the base address supposed to be
10:25:35 <nyc> mobile_c: The vaddr is the one userspace loaders pay attention to.
10:25:47 <nyc> mobile_c: It varies with the arch ABI.
10:26:35 <nyc> mobile_c: Most things follow an SVR4-ish ABI for program loading and user address space layout.
10:27:05 <nyc> e.g. http://www.sco.com/developers/devspecs/abi386-4.pdf
10:29:19 <nyc> mobile_c: e.g. p. 47 has ```Although the exact boundary between the reserved area and a process depends on the system’s configuration, the reserved area shall not consume more than 1 GB of the address space. Thus the user virtual address range has a minimum upper bound of 0xc0000000```
10:30:53 <nyc> mobile_c: p.48 specifies that .text begins at 0x8048000
10:31:13 <nyc> mobile_c: And (Linux violates this) that the stack is below the text.
10:32:55 <mobile_c> so.....
10:33:06 <nyc> I guess it's not gospel now that PIE etc. are gaining popularity.
10:33:09 <mobile_c> what does all that mean for
10:33:11 <mobile_c> library[library_index]._elf_program_header[lowest_idx].p_paddr = 0x000000200e78
10:33:12 <mobile_c> library[library_index]._elf_program_header[lowest_idx].p_vaddr = 0x000000200e78
10:33:37 <mobile_c> in relation to " The base address is the difference between the truncated memory address and the truncated p_vaddr value."
10:33:46 <nyc> mobile_c: I would probably need a section name to make sense of that.
10:34:36 <mobile_c> PT_LOAD
10:34:38 <mobile_c> https://paste.pound-python.org/show/FqyKhFGK0NzjZtjxOfE2/
10:35:05 <nyc> mobile_c: It looks like the permissions aren't distinct from adjacent sections because if it were the first one with its permissions it would be page-aligned.
10:35:45 <nyc> mobile_c: Are you trying to fish ELF info out of a running program?
10:36:27 <mobile_c> no
10:36:49 <nyc> mobile_c: So you're parsing an ELF file?
10:37:01 <mobile_c> yes
10:37:38 <nyc> mobile_c: Where did the quote " The base address is the difference between the truncated memory address and the truncated p_vaddr value." come from?
10:38:50 <nyc> x86-64 SVR4/ELF standard?
10:38:58 <mobile_c> https://paste.pound-python.org/show/UxrFuk2jbAzSLIdHEocp/
10:39:11 <mobile_c> yes
10:39:18 <mobile_c> see Base Address in the code
10:39:34 <mobile_c> An executable or shared object file's base address is calculated during execution from three values: the virtual memory load address, the maximum page size, and the lowest virtual address of a program's loadable segment. To compute the base address, one determines the memory address associated with the lowest p_vaddr value for a PT_LOAD segment. This address is truncated to the nearest multiple of the maximum page size. The corresponding p_vaddr value
10:39:35 <mobile_c> itself is also truncated to the nearest multiple of the maximum page size. The base address is the difference between the truncated memory address and the truncated p_vaddr value.
10:40:54 <nyc> http://www.sco.com/developers/gabi/2003-12-17/ch5.pheader.html
10:41:06 <mobile_c> or specifically...
10:42:38 <mobile_c> Tool Interface Standard (TIS) Executable and Linking Format (ELF) Specification Version 1.2.pdf > Book III: Operating System Specific (UNIX System V Release 4) > 2. Program Loading and Dynamic Loading > Program Header > Base Address
10:44:26 <mobile_c> which can be found here
10:44:46 <mobile_c> http://refspecs.linuxbase.org/elf/elf.pdf
10:45:01 <nyc> mobile_c: I think it's talking about the address differences between variables in different segments of shlibs.
10:45:41 <nyc> mobile_c: The whole shlib might move, but the relative positions of the text and data and rodata remain the same.
10:46:03 <nyc> mobile_c: That's what I'm getting out of it at the moment.
10:46:52 <nyc> ```On some platforms, while the system chooses virtual addresses for individual processes, it maintains the relativeposition of one segment to another within any one shared object.```
10:48:11 <nyc> mobile_c: For a shlib it's basically talking about where the first section of it is loaded.
10:50:39 <mobile_c> https://paste.pound-python.org/show/tF1emhGg65gLcSPg2kor/ *
10:50:47 <nyc> mobile_c: Basically for a shlib the vaddrs all start from zero and when a shlib gets loaded up they get their real starting addresses added in somewhere.
10:52:02 <nyc> s/somewhere/at runtime/
10:52:28 <mobile_c> assume i am executing...
10:52:52 <mobile_c> void* DT = dlopen("./DT_INIT.so");
10:53:43 <nyc> mobile_c: The handle dlopen() returns is not necessarily the address of the shlib in memory.
10:57:06 <mobile_c> as this is my current program https://github.com/mgood7123/universal-dynamic-loader/blob/master/loader/readelf_.c
10:57:22 <klange> it's an opaque pointer to a structure you use yourself to figure out where things are
10:58:18 <nyc> mobile_c: I can't quite absorb the whole of it at a glance. What are you trying to do?
10:58:31 <nyc> mobile_c: Are you writing your own ld.so?
10:58:42 <mobile_c> klange: oh, i just return a char string lmao
10:59:11 <mobile_c> char string for mine lmao *
10:59:45 <mobile_c> void * dlopen(const char * cc) { get_needed(cc); return dlopen_(cc); }
11:00:09 <mobile_c> void * dlopen_(const char * cc) { ... return cc; }
11:01:55 <mobile_c> https://github.com/mgood7123/universal-dynamic-loader/blob/3c28f11bc6d9e56796a1ca5eafa1d5023aa7a46f/loader/readelf_.c#L3820
11:02:19 <klange> https://github.com/klange/toaruos/tree/master/linker
11:02:26 <nyc> I'm trying to figure out how I can give the sun4u OpenBIOS a kernel so I can actually testboot anything.
11:02:49 <nyc> It's surprisingly difficult to dredge up the documentation for it.
11:03:03 <mobile_c> anyway
11:03:34 <mobile_c> what would the base address be in the context of dlopen("./DT_INIT.so");
11:04:04 <klange> whatever you want as long as it makes sense in the context of the available virtual memory?
11:04:30 <mobile_c> klange: ._.
11:04:35 <klange> calculate the size needed to load the library, allocate space for it through whatever mechanism you feel is appropriate - probably mmap, because you're on Linux and can do *real* shared libraries.
11:04:47 <klange> and then your base address is where you load stuff
11:05:06 <mobile_c> nyc:
11:05:10 <klange> If you want to skip actually sharing your shared libraries, you can literally just valloc the necessary space.
11:05:18 <nyc> mobile_c: ?
11:05:19 <klange> and bam, there's your base address
11:05:50 <mobile_c> klange: the base address is NOT the same as the address the .so gets mmapped to
11:06:35 <klange> it's the addres for which the offsets are aligned
11:07:10 <nyc> http://www.helenos.org/releases/HelenOS-0.8.0-sparc64-ultra.iso <--- looks like ISO9660
11:08:00 <klange> nyc: On a 2.5MB CD image? Of course it's ISO 9660.
11:08:42 <mobile_c> klange: if that is so then ASSUMING according to klange the base address is THE SAME as the mapping address, WHY ARE MY DT_INIT_ARRAY indexes NULL
11:08:58 <klange> do they have relocations?
11:09:01 <mobile_c> yes
11:09:24 <nyc> I think there are more details to it.
11:10:14 <nyc> There's an ELF executable but also some things that look suspiciously like boot code placed at magic sectors.
11:11:26 <mobile_c> actually no, if i SET my base_address to my mpping_address i get this
11:11:46 <mobile_c> https://paste.pound-python.org/show/ll8PefQILdPbW2316Itl/
11:12:01 <klange> nyc: I'm sure the helenos source repo has the genisoimage recipe for building something like this
11:12:25 <mobile_c> in which the relocation is this
11:12:27 <nyc> klange: It looks like it uses silo.
11:13:05 <nyc> I used to use sparc64 boxen all the time.
11:13:11 <nyc> I never booted off disk.
11:13:16 <nyc> I always booted off the network.
11:14:59 <nyc> It's pretty simple. You have to do a round trip through a known-good kernel to put a new possibly-good kernel where it can be found on disk. If you cross compile and tftpboot you can just testboot possibly-good kernels attempting bugfixes etc. rapid fire.
11:17:43 <klange> https://i.imgur.com/VOPsvLS.png *shrug*
11:18:01 <zid> Right, how do I get qemu pingable, bochs is pissing me off with its idiosynchracies and I wanna try qemu
11:18:28 <zid> the device gets created inside the guest, but idk what device to ping from the host side
11:18:45 <zid> adjusting tap0 with ifconfig makes qemu print messages, but giving it an ip and pinging it seems to do nothing
11:19:15 <mobile_c> https://paste.pound-python.org/show/esuS4mRnx1aHN6IMZcll/ i get that
11:19:21 <nyc> qemu-system-sparc64: Invalid SMP CPUs 4. The max CPUs supported by machine 'sun4u' is 1
11:19:24 <nyc> That makes me cry.
11:19:29 <mobile_c> for ./files/libstring.so
11:19:31 <zid> netdev tap,ifname=tap0,id=net0 -device e1000,netdev=net0 is apparently my command line
11:19:38 <zid> (ancient script I found laying around to start qemu for my code)
11:19:59 <klange> zid: are you pinging the guest's IP or the tunnel's IP?
11:20:28 <zid> klange: with bochs I can just ping 192.168.200.>1 (2-255) after assining 192.168.200.1 to tap0
11:20:32 <zid> with qemu that does nothing
11:21:42 <mobile_c> odlly it works for dt_init.so but it doesnt work for anything else
11:21:58 <zid> bochs for some reason is being super annoying and there's a several second delay between the ping and the guest seeing the packet and updating the screen, idk where in the stack the pause is but I'd love to try qemu instead
11:22:33 <mobile_c> so yea... the base address must not be the mapping address itself
11:23:18 <mobile_c> in which results in
11:23:19 <mobile_c> get_dynamic_entry(library[library_index].dynamic, DT_INIT_ARRAYSZ)->d_un.d_val / sizeof (Elf64_Addr) = 3, ((Elf64_Addr *) (library[library_index].base_address + init_array->d_un.d_ptr))[0] = 0x00001000368d, ((Elf64_Addr *) (library[library_index].base_address + init_array->d_un.d_ptr))[1] = 0x0000100026a0, ((Elf64_Addr *) (library[library_index].base_address + init_array->d_un.d_ptr))[2] = 0x00001000452e
11:23:44 <mobile_c> if the mapping address is used as the base address
11:24:34 <mobile_c> in which
11:24:35 <mobile_c> library[library_index].base_address = 0x7f9e10000000
11:24:36 <klange> If a phdr says its vaddr is x, it should be in memory at base + x.
11:25:13 <klange> Done. End of story. Nothing more to base addreses than that. You can pick whatever you want for them as long as you don't collide with other stuff.
11:27:19 <mobile_c> get_dynamic_entry(library[library_index].dynamic, DT_INIT_ARRAYSZ)->d_un.d_val / sizeof (Elf64_Addr) = 3, ((Elf64_Addr *) (library[library_index].base_address + init_array->d_un.d_ptr))[0] = 0x0x7f9ff000368d, ((Elf64_Addr *) (library[library_index].base_address + init_array->d_un.d_ptr))[1] = 0x0x7f9ff00026a0, ((Elf64_Addr *) (library[library_index].base_address + init_array->d_un.d_ptr))[2] = 0x0x7f9ff000452e *
11:27:42 <mobile_c> or something, idk if i need to print it using %x or %p
11:28:56 <mobile_c> klange: so its this?
11:29:26 <mobile_c> library[library_index].base_address = library[library_index].mapping_start + truncated_virtual_address;
11:30:44 <klange> The base address should be the virtual address to which vaddr for a phdr is added to get the final virtual address of the phdr in memory...
11:31:25 * zid tries to mess around with bridges instead
11:31:32 <mobile_c> so
11:31:32 <nyc> Ugh, silo isn't portable to other arches.
11:31:33 <mobile_c> library[library_index].base_address = library[library_index].mapping_start + library[library_index]._elf_program_header[lowest_idx].p_paddr + library[library_index]._elf_program_header[lowest_idx].p_vaddr;
11:31:35 <mobile_c> ?
11:31:41 <klange> no
11:31:44 <klange> you have that completely backwards
11:32:05 <nyc> I am in for a world of toolchain hurt.
11:32:14 <klange> *(base_address + vaddr) = shit goes here
11:32:32 <nyc> HelenOS booted up and got into a GUI though, whatever that's worth.
11:32:38 <klange> instead of thinking about the mapping, think only about where the phdr contents end up in memory
11:32:48 <mobile_c> so what is base_address itself
11:32:53 <klange> whatever you want it to be
11:32:58 <klange> you have to pick where to put the library
11:33:12 <mobile_c> as u CANT do base_address = *(base_address + vaddr);
11:33:13 <klange> that's the fundamental concept behind these kinds of *relocatable libaries*, you have to relocate them
11:33:19 <klange> that literally doesn't make sense
11:33:27 <klange> you pick one base address for each library
11:33:52 <mobile_c> imma just use the mapping address + vaddr
11:34:02 <klange> where are you getting vaddr from?
11:34:17 <klange> Your library has ONE base address but multiple vaddr for each PT_LOAD PHDR.
11:34:36 <mobile_c> thelowest one?
11:34:42 <mobile_c> the lowest one? *
11:35:22 <klange> You pick a base address. You go through each PHDR with PT_LOAD and load it - by mapping it or whatever, so that its contents are in memory at library.base_address + phdr.vaddr
11:35:49 <klange> Then you use that base_address for any calculation that needs a base address.
11:36:24 <mobile_c> so the base address is the final base address of the PT_LOAD that gets loaded last?
11:36:50 <klange> NO
11:36:51 <mobile_c> as it would keep overwriting the base address for each PT_LOAD loaded
11:37:00 <klange> There is *one* base address, you pick it at the start!
11:37:02 <mobile_c> as it is assigned on every PT_LOAD
11:37:04 <klange> You don't change it!
11:37:06 <klange> NO
11:37:08 <klange> omg
11:37:12 <klange> You are not listening to a word I am saying.
11:37:30 <mobile_c> then WHICH phdr.vaddr do i use
11:37:44 <klange> You do not use phdr.vaddr in the calculation of base address.
11:37:49 <klange> You have this entirely backwards.
11:38:02 <klange> You pick the base address. It can be ANYTHING as long as you can fit the loaded portions of the library into memory after it.
11:38:11 <klange> I do a calculation when loading to find out how much space is needed.
11:39:15 <klange> Then you load the PT_LOAD PHDR contents at their respective `base_address + phdr.vaddr`. There's usually two of them with different memory permissions.
11:39:41 <klange> Then you perform relocations. The library only ever has one base address, it is assigned before loading things, and it never changes.
11:40:08 <klange> There is no "mapping address", you are thinking about things from the wrong perspective with that. You do not load the whole file into memory and start tinkering with it.
11:40:35 <klange> You read the headers and load or map what you need to.
11:40:44 <klange> That's what the PHDRs are there for.
11:41:10 <zid> you *can* load the entire file into memory, and then make a second set of mappings, then unmap the originals
11:41:26 <zid> If you actually look at a linux process, the ELF header is in there, same on windows with the PE/MZ
11:41:39 <nyc> Maybe I should do ARM/aarch64 next instead of SPARC so I can make faster progress (plus odds are more people here would be able to basically handhold me through the whole process if I hit any hiccups along the way).
11:41:43 <zid> that's a weird optimization though
11:42:16 <klange> Generally one of your PT_LOADs has offset=0 anyway.
11:42:24 <klange> So you get the header for free™.
11:42:39 <nyc> zid: It's basically to avoid wasting disk space to pad to an alignment boundary.
11:42:43 <klange> It's the *other* PT_LOADs that won't be aligned.
11:42:48 <klange> LOAD 0x002394 0x00003394 0x00003394 0x000dc 0x152f4 RW 0x1000
11:43:04 <klange> See this load for a 32-bit SO? The offset into the file and the virtual address in memory are different!
11:43:35 <klange> You can't just stick the file somewhere and call it done. Those two loads need to be mapped into different parts of memory, they are not contiguous as they are in the file.
11:45:06 <nyc> aarch64 qemu looks like it can do at least 4x SMP.
11:45:15 <zid> well I got the bridge up, still no packets, hrmph
11:45:29 <zid> wonder if there's some config difference on the guest side etween qmu and bochs I am not accounting for
11:45:46 <zid> or maybe an emulation difference in qemu/bochs meaning I am not getting irqs or packets
11:46:39 <mobile_c> so if the .so is mmapped at 0x7f0259012000 does that become the base address
11:46:50 <klange> you do not map the .so into memory at anything
11:47:07 <mobile_c> if 0x7f0259012000 is returned by mmap *
11:47:08 <klange> you map *parts* of the SO at *different locations* as specified by the PT_LOAD PHDRs and the virtual addresses.
11:47:31 <mobile_c> eg
11:47:32 <mobile_c> mmap (NULL, library[library_index].len, PROT_READ, MAP_PRIVATE, fd, 0);
11:47:39 <klange> sure
11:47:55 <mobile_c> does the address returned by that become the base_address
11:47:58 <klange> if you're just mapping regular memory, it's the address where that chunk is
11:48:03 <ybyourmom> Stop colluding
11:48:32 <zid> Time to debug code that works in bochs but not qemu, sounds fun heh
11:49:05 <klange> Generally, you'd want to keep track of the virtual addresses yourself.
11:49:30 <klange> You can easily figure out how much space a library needs by going through its PT_LOAD PHDRs and finding the one with the highest vaddr+memsize
11:49:43 <mobile_c> eg https://github.com/mgood7123/universal-dynamic-loader/blob/master/loader/readelf_.c#L578
11:49:46 <klange> (Also need to consider the alignment)
11:50:13 <klange> no
11:50:16 <klange> stop mapping the file
11:50:17 <klange> don't do that
11:50:27 <klange> READ the headers, the old fashioned way
11:50:34 <mobile_c> if i dont map the file then how do i read it ._.
11:50:38 <klange> the old fashioned way?
11:51:13 <klange> If you want to you can mmap it somewhre, but that has nothing to do with the address you are loading it to.
11:51:29 <mobile_c> so what? library[library_index]._elf_header = (Elf64_Ehdr *) read(fd, max_len,); ?
11:51:38 <mobile_c> or something?
11:51:42 <klange> Sure, whatever.
11:52:06 <klange> That's not how read works, but whatever.
11:52:13 <mobile_c> meh
11:53:26 <klange> https://github.com/klange/toaruos/blob/master/linker/linker.c#L206
11:53:46 <klange> then I read the phdrs once to find out how much space the library will need https://github.com/klange/toaruos/blob/master/linker/linker.c#L233
11:54:06 <zid> -if(!r) { +if(r < sizeof (Elf32_Header)) {
11:54:15 <klange> then I load it by reading it into memory; you have Linux, you can mmap the regions of the file https://github.com/klange/toaruos/blob/master/linker/linker.c#L273
11:54:32 <mobile_c> btw this is the read_fast_verifyb function https://github.com/mgood7123/universal-dynamic-loader/blob/master/loader/libstring.c#L2127
11:55:39 <klange> zid: with fread(...,sizeof(Elf32_Header),1,...) should actually be if(r<=0), as it should return 1 for a full header having been read
11:56:17 <zid> oh right you actually use the nmemb and ncount things
11:56:24 <zid> You very rarely see that, read straight over it
11:57:39 <klange> the secret to my C style is that in my years of writing C I haven't really read anyone elses'.
11:57:43 <zid> hehe
11:57:49 <zid> I'm pretty guilty of the same
11:58:05 <mobile_c> which is used as per
11:58:06 <mobile_c> read_fast_verifyb(library[library_index].array, library[library_index].len, &library[library_index].mapping_start, span, library[library_index]._elf_program_header[library[library_index].First_Load_Header_index], library[library_index]._elf_program_header[library[library_index].Last_Load_Header_index]);
11:58:20 <zid> I'm fairly sure at one point I used an implementation that (incorrectly) failed if the element size was funny
11:58:25 <zid> but doing it all as bytes always works
11:58:31 <mobile_c> ohhhhhhh thats how i aquire the mapping_start variable
11:59:03 <mobile_c> (cus i couldnt find an assigment for it, eg 'mapping_start = ')
12:02:50 <mobile_c> klange: where do i aquire mpfr headers from
12:03:24 <klange> libmpfr-dev?
12:03:31 <mobile_c> ok
12:03:48 <mobile_c> so i need this right?
12:03:49 <mobile_c> sudo apt install xorriso gnu-efi gmp* libmpfr* mpc*
12:03:56 <klange> What are you building?
12:04:10 <mobile_c> neon@neon-X555BA:~/git/universal-dynamic-loader/loader/resources/toaruos$ make
12:04:11 <mobile_c> MPC headers are required to build GCC cross-compiler
12:04:14 <mobile_c> ;-;
12:04:22 <klange> why are you building toaruos? ;)
12:04:30 <klange> use the docker build, tbh
12:04:46 <mobile_c> ok so its libmpc
12:04:50 <mobile_c> not mpc
12:05:09 * mobile_c builds the toolchain
12:05:15 <klange> whyyyy
12:05:28 <klange> just use the docker build it takes two minutes to download the image vs. a half hour to build gcc
12:05:29 * mobile_c even though i have an existing one installed
12:05:46 <mobile_c> klange: i wanna see what ur linker can do :P
12:05:52 <klange> just download the CD image then
12:06:00 <klange> and install gcc locally in the VM
12:06:16 <mobile_c> more specifically if it is compatible with gnu libc xP
12:06:27 <klange> uh, you will not be able to figure that out
12:06:32 <mobile_c> (not the cross compiled tourus version)
12:06:55 <klange> well my linker only works with toaruos because it uses some special syscalls directly
12:06:59 <mobile_c> as i assume that one would differ from standard gnu libc
12:07:01 <klange> so you can't expect it to work on Linux
12:07:07 <klange> there is no ported glibc
12:07:11 <klange> so you can't figure it out from there either
12:07:26 <nyc> Ouch. It looks like the death knell has sounded for SPARC. =(
12:07:28 <mobile_c> oh
12:08:17 <klange> glibc is crazy difficult to port to a new environment, it was not designed for it
12:08:27 <klange> toaruos used newlib for a while until I went in and started building my own libc
12:08:33 <mobile_c> byte my linker CAN load libraries and execute functions sucesfully, it just cant initialize the libs
12:08:35 <mobile_c> btw*
12:08:51 <nyc> Oracle buyout + layoffs of the vast majority of SPARC hardware and Solaris staff. Fujitsu is still doing SPARC, but it's very niche.
12:08:59 <mobile_c> eg libc requires initialization to be able to execute the puts function without it hanging
12:09:19 <mobile_c> write and strlen work with without initialization
12:09:24 <mobile_c> work fine*
12:09:36 <mobile_c> puts and printf dont however
12:09:42 <klange> puts needs the std* stuff to be set up already
12:10:22 <mobile_c> also it can also load musl ^-^
12:10:29 <nyc> mobile_c: puts(s) is fputs(s, stdout) so stdout needs to be initialized i.e. allocated with all of the FILE affairs initialized.
12:10:32 <klange> I use one constructor in my libc, and it does that - sets up stdio buffers.
12:10:45 <klange> It also does some processing on environ and checks if __LIBC_DEBUG is set.
12:11:04 <mobile_c> tho it cant (yet) dlopen recursively
12:11:13 <mobile_c> i think
12:11:21 <mobile_c> im not sure
12:11:40 <mobile_c> i think it tries to?
12:11:50 <mobile_c> i havnt tried in a while
12:12:03 <mobile_c> been focusing on trying to get the initialization to work
12:13:04 <mobile_c> btw is ur linker standalone
12:13:16 <mobile_c> eg not integrated with glibc
12:13:19 <klange> Everything in ToaruOS is standalone.
12:13:21 <mobile_c> like gnu's ld.so is
12:13:25 <nyc> I'll save getting the SPARC tools going for later on today. (Much of silo appears to be written in SPARC asm.)
12:13:33 <klange> But the linker uses some libc functions, which it links in statically.
12:13:38 <mobile_c> like gnu's ld.so is integrated heavily with glibc *
12:13:52 <nyc> klange: Are you still considering a compiler? Or ports to other CPU's?
12:14:08 <klange> still considering a compiler, ports to other architectures not high on my list except x86-64
12:14:17 <klange> kinda busy with buying a house at the moment
12:14:28 <nyc> klange: In Japan?
12:14:31 <klange> Yep.
12:14:35 <mobile_c> eg can ur linker be compiled with almost any variant of libc and still work
12:14:45 <nyc> klange: Expensive real estate.
12:14:54 <zid> japan is cheaaaap
12:14:58 <mobile_c> (provided that libc provedes the required functions and work as expected)
12:15:02 <klange> Eh, not compared to big US cities.
12:15:11 <zid> or most decent europeen cities
12:15:16 <nyc> klange: Okay, I guess it depends on where.
12:15:19 <klange> Found a nice place with good transit access for ¥35m
12:15:36 <mobile_c> (as i suspect some #ifdef's will be required for compatibility with different libc's )
12:15:36 <zid> japanese houses are made of paper maché, makes em cheap
12:15:38 <klange> Fits all my requirements, new construction, 100m²
12:15:44 <klange> zid: not modern construction :P
12:15:46 <mobile_c> when compiling with*
12:15:50 <zid> :D
12:16:22 <mobile_c> also as far as i know, my linker CANNOT be compiled into a static executable
12:16:25 <klange> mobile_c: my linker only works with my OS because it does some syscall stuff directly that only my kernel has
12:16:30 <nyc> I guess if I play up the SPARC angle, there is Fujitsu, but I've never seriously considered Japan.
12:16:51 <mobile_c> (i think, that may be another project im thinking of)
12:17:50 <mobile_c> yea i think thats for my userland-exec port that cannot be compiled statically
12:18:20 <nyc> I'll try to get some rest before my afternoon appointment.
12:18:57 <mobile_c> not sure. as my linker is meant to be preloadable so i do not think a static archive can be preloadable
12:19:25 <klange> I build mine as a static binary.
12:19:31 <mobile_c> yea im just gonna assume the static part os for me exec project instead of my linker
12:19:36 <mobile_c> is*
12:19:38 <mobile_c> my*
12:20:30 <mobile_c> which im not sure what the exec is meant to do as it been years since i visited it
12:21:34 <klange> I'm going to go sit in bed and watch twitch streams and probably call it a night.
12:21:38 <mobile_c> think it might be something to do with being able to execute files from within a self extracting executable directly without actually extracting the files or something
12:22:02 <mobile_c> like a virtual chroot inside the executable itself or something
12:22:14 <mobile_c> not sure
12:23:23 <mobile_c> anyway
12:23:32 * mobile_c waits for glibc to compile
12:25:18 <nyc`> I'm ambivalent about POSIX etc. On the one hand, I can't really NIH the entire world. On the other hand, C and UNIX are relatively offensive to modern or at least my sensibilities.
12:25:25 <mobile_c> btw can your linker load 64 bit ELF's or onyl 32 bit
12:25:32 <mobile_c> only*
12:25:33 <klange> only 32 bit, I don't have a 64-bit target
12:25:37 <mobile_c> ok
12:25:39 <klange> but the principles are all the same
12:27:39 <mobile_c> ok
12:28:43 <mobile_c> why 'uintptr_t base_addr = 0xFFFFFFFF;' ?
12:29:51 <mobile_c> also my linker can load resonably more types of functions than gnu or musl can, such as local functions and nested function
12:30:10 <mobile_c> (eg it is not limited to just global functions)
12:30:39 <klange> Did you read the function that line was used in?
12:30:50 <klange> (It has nothing to do with the base address we have been talking about)
12:30:56 <mobile_c> provided og course the local and nested functions do not depend on values created from other functions
12:30:58 <mobile_c> of*
12:31:13 <klange> It's just trying to find the size of the library based on the lowest and highest virtual addresses it touches
12:31:22 <mobile_c> as then it would probably segfault due to uninitialized variables
12:32:28 <mobile_c> would this be one of them
12:32:29 <mobile_c> sysfunc(TOARU_SYS_FUNC_MMAP, args);
12:33:28 <klange> for args={addr,size}, that requests memory be allocated at virtual address `addr` for `size` bytes
12:33:44 <mobile_c> ok
12:33:57 <klange> you could feasibly replace that and the following fread with an mmap of the same region of the file into that address
12:34:02 <mobile_c> would it be difficult to port the linker to a normal linux kernel?
12:35:18 <klange> I don't know, I have not had any interest in doing so.
12:35:25 <mobile_c> ok
12:36:25 <mobile_c> so the only functions are
12:36:27 <mobile_c> sysfunc(TOARU_SYS_FUNC_MMAP, args); and sysfunc(TOARU_SYS_FUNC_SETHEAP, args);
12:36:29 <mobile_c> ?
12:44:20 <mobile_c> "as well as not handling symbol resolution correctly." what does that mean o.o
12:45:38 <klange> the symbol table doesn't track where symbols came from so it can't do NEXT properly
12:45:52 <klange> not super important until you get into weird preload libraries that override stuff
12:46:46 <mobile_c> ok
12:47:16 <nyc`> I would probably actually want a low-level continuation-passing-style language instead of C that actually has a module system. And I'd want a totally async request issuing and completion notification "syscall" interface. But I can't write literally the whole of userspace from scratch, and I really only have innovations to bring to the kernel.
12:47:33 <klange> why not?
12:48:04 <klange> toaruos is unix-y, but it's also all from scratch; it would have been *easier* if I wasn't also trying to be compatible enough to gcc, python, etc.
12:48:50 <mobile_c> what would the PROT be for sysfunc(TOARU_SYS_FUNC_MMAP, args);
12:49:05 <mobile_c> void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset);
12:49:28 <klange> it's not setting any
12:49:41 <klange> if you want that you need to figure it out from the phdr and do it yourself
12:49:56 <klange> missing kernel interfaces to set protections atm, so don't have that in my linker
12:52:18 <mobile_c> ok
12:52:34 <nyc`> klange: I think getting things like GUI's and browsers and databases and such is made more difficult by not having syscall etc. semantics that map to UNIX in any 1:1 way.
12:53:36 <klange> unless you're a large team at Google, I wouldn't expect you to get a browser even if you did have a Unix-like API, so maybe discount that one.
12:54:36 <nyc`> klange: If you say "nothing blocks anymore" then nothing ports and a lot of things will need total redesigns.
12:56:43 <nyc`> klange: I think there are rather stringent limits to even how much of the kernel can be NIH'd and userspace is so much vaster.
12:57:17 <klange> "I want to do a thing but also I don't want to do the thing because it's not feasible to do the thing."
12:57:24 <klange> This seems to be your perspective on several topics.
12:58:37 <mobile_c> so it would be this?
12:58:39 <mobile_c> mmap((char *)(base + phdr.p_vaddr), phdr.p_memsz, prot_from_phdr(phdr.p_flags), MAP_FIXED|MAP_ANONYMOUS, 0, 0);
12:59:15 <klange> *shrug*
12:59:19 <klange> probably
12:59:52 <mobile_c> ad TOARU_SYS_FUNC_SETHEAP ?
12:59:55 <mobile_c> and*
01:00:02 <nyc`> klange: Given greater resources i.e. many minions I could NIH large swaths of userspace, though still with major holes in it barring truly vast numbers.
01:00:10 <mobile_c> would that just be setbsrk or something
01:00:18 <klange> nyc`: google says hi
01:00:27 <klange> mobile_c: *shrug*
01:00:39 <mobile_c> like sbrk(end_addr);
01:01:11 <klange> sbrk takes an increment, maybe brk would work
01:01:29 <nyc`> klange: Google wasn't interviewing me for management or even PM / team lead, and they turned me down every time.
01:02:25 <klange> same
01:11:40 <nyc`> I had some situations where I got denied bodies in the 2000's. I basically got told to do things solo, which meant 120+ hour weeks and no vacation and so on.
01:13:02 <mobile_c> ok so i tried the linker and i get this
01:13:03 <mobile_c> https://paste.pound-python.org/show/nfQUTSxfXO5MXj17F6lc/
01:13:06 <mobile_c> klange:
01:13:47 <klange> And /bin/ls is...?
01:13:57 <klange> I'm not sure how far I can help you with trying to prot my linker to linux.
01:14:56 <mobile_c> neon@neon-X555BA:~/git/universal-dynamic-loader/loader/resources/toaruos/linker$ file /bin/ls
01:14:57 <mobile_c> /bin/ls: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=9567f9a28e66f4d7ec4baf31cfbf68d0410f0ae6, stripped
01:15:09 <klange> duh?
01:15:16 <klange> you already asked me if it supported 64-bit and I said no
01:15:57 <klange> I'm surprised it doesn't shit itself a lot earlier
01:17:47 <zid> klange: What networking setup were you using to test yours, out of interest
01:17:54 <nyc`> klange: Anyway, I've got my hands full just trying to get hello world and other basic bootstrapping going on a few different arches.
01:18:02 <zid> I know you have e1000, I found your driver at one point on google :P
01:18:07 <mobile_c> ok
01:18:15 <mobile_c> ill try to compile a 32 bit binary
01:18:28 <klange> zid: I mostly use the default setup which is slirp
01:18:40 <zid> how does that look from the host side?
01:18:54 <klange> it doesn't
01:19:03 <zid> oh, it's a loopbacky thing?
01:19:22 <klange> slirp makes TCP / UDP connections like a regular user application on the host, and makes them look like raw packets to the guest
01:19:34 <zid> right that's what I thought, but you said not that
01:19:35 <klange> you can't do some low-level things because slirp won't emulate them
01:20:07 <klange> I've done the tunenl stuff with vmware and virtualbox; vmware it's the default and seemingly only working setup
01:20:21 <zid> oh I haven't considered vmware, maybe I should try that :D
01:20:44 <nyc`> If I get to where I can even run userspace, then I might deliberate more about how much userspace to NIH.
01:22:18 <klange> A GUI and a text editor are all anyone should need.
01:26:57 <nyc`> Compilers, databases, web browsers, there are big things in userspace.
01:27:32 <izabera> why the hell would you put a compiler in userspace
01:27:46 <klange> because that's usually where you put compilers?
01:28:04 * izabera only runs gcc on ring 0
01:28:04 <zid> what if your userspace is exclusively non-binary
01:28:18 <zid> brainfuck userspace, compiler in kernel
01:28:32 <izabera> was that an enbi joke
01:28:45 <nyc`> They're rare to put in the kernel, though I suppose Symbolics did.
01:33:31 <mobile_c> klange: https://paste.pound-python.org/show/UQgYieqGS1Nuhr4t40r4/
01:34:04 <klange> mobile_c: add more debug output? I'm not going to babysit you through porting my linker to linux
01:34:14 <klange> mobile_c: it's getting late here and I have work tomorrow
01:35:08 <nyc`> UNIX itself was originally a smaller project than some of us are taking on.
01:35:46 <mobile_c> mmap(NULL, 1896, PROT_READ|PROT_EXEC, MAP_FILE|MAP_FIXED, 0, 0) = -1 EPERM (Operation not permitted)
01:36:52 <klange> unix was just a fancy text processing system
01:38:13 <nyc`> I think it wasn't even originally multitasking, hence Unix vs. Multics.
01:45:38 <nyc`> There were other OS's out there that had more interesting ideas than being Multics knockoffs. Multics itself, for that matter.
01:56:14 <mobile_c> klange: why do i get
01:56:16 <mobile_c> (base + phdr.p_vaddr) = (nil)
01:56:29 <klange> idk you did something wrong somewhere *shrug*
01:57:02 <klange> I said already I'm not going to babysit you through porting my linker to Linux
01:57:15 <mobile_c> base = 0x00000000000000
01:57:17 <mobile_c> phdr.p_vaddr = 0x00000000000000
01:59:32 <mobile_c> apparently it seems to be partially cus of
01:59:34 <mobile_c> uintptr_t end_addr = object_load(main_obj, 0x0);
02:00:09 <nyc`> Tanenbaum really did some wonders with Amoeba. VMS was rather impressive. Symbolics did its Lisp thing. Xerox did its Smalltalk thing. Etc. Unix' VFS is its greatest fault. Sockets are a mess for networking. I think Unix memory management leaves a fair amount to be desired as well.
02:03:20 <klange> mobile_c: you may need to assign a base address for the main object (the binary)
02:04:04 <klange> my binaries already load to a specific address
02:04:50 <klange> but a quick check of an -m32 linux binary has its low vaddr at 0
02:04:55 <klange> so you'll need to assign one to get things started
02:04:58 <klange> I'm going to bed now
02:05:47 <nyc`> 0 is not usually used for absolute code.
02:06:04 <klange> linux stuff is probably all set up for ASLR
02:06:18 <klange> so it has no high load address and you need to pick one
02:06:32 <nyc`> ASLR?
02:07:29 <nyc`> Security by obscurity.
02:08:08 <mobile_c> as in yours is assigned to 0x000040000000 ?
02:08:31 <klange> you have way too many digits there
02:08:48 <mobile_c> p_paddr: 0x000040000000
02:09:04 <mobile_c> or 0x40000000
02:09:05 <klange> you have way too many digits there
02:09:25 <mobile_c> heh, i just do fprintf(stderr, "\t\tp_paddr: %014p\n", phdr.p_paddr);
02:10:01 <klange> why 14?
02:10:33 <mobile_c> cus 64 bit lol
02:13:08 <nyc`> Usually it's 16 nybbles.
02:19:32 <klange> Anyway, good luck with that, and good night.
02:32:46 <nyc`> I wish there were sparc64 firmware for qemu that would tftpboot. Maybe one of the alternatives to OpenBIOS if I can get binary images and if qemu's simulation is good enough.
02:34:44 <nyc`> The Niagara target won't even start up without what are likely closed source firmware binaries, so that's what I'm probably really up against.
02:38:17 <nyc`> It's sort of sad that SPARC is massively multithreaded and the minimum packages are something like 32 core 256 thread and the simulator can't do SMP (probably from lack of docs on how it works).
02:39:50 <nyc`> No one seems to have come out of the recent mass layoffs with the relevant documents, so it's probably going down with the ship.
02:40:26 <nyc`> When DEC died similar happened to Alpha.
02:45:49 <nyc`> I think the entire threading affair was originally DEC 21364/21464 material. I feel responsible. If I had done a better job making IA64 work, x86(-64) wouldn't have killed everything.
02:57:21 <zhiayang> \o/, system calls
02:58:43 <zhiayang> though, is there something better than this macro disaster: https://hastebin.com/ufohunoboc.pl
03:01:53 <nyc`> Ring buffer for out-of-order completion?
03:02:07 <zhiayang> ?
03:05:12 <nyc`> Synchronous system calls suck.
03:07:20 <zhiayang> i mean
03:07:37 <zhiayang> sure, but i need to start somewhere, right?
03:09:25 <nyc`> zhiayang: If you're beyond hello world on only one arch, you're ahead of me.
03:10:34 <zid> Anyone experienced bochs' network light flashing green a few times, then a few seconds later flashing red and the irq shows up?
03:31:29 <nyc`> My grand plan is really mostly to do with virtual memory. I don't have original ideas about much else. I just have other people's ideas that I happen to like for a variety of other affairs.
03:48:49 <nyc`> There are things it would be nice to do for userspace, but my original idea is just for VM. So userspace is beyond the scope of what I have that's worth demonstrating even if it makes my eyes bleed.
03:51:16 <nyc`> Well, there are exceptions for hooks to show off my VM affairs.
03:52:33 <zhiayang> so your endgoal is to demonstrate your vm idea on every conceivable arch with an mmu?
03:53:37 <nyc`> zhiayang: No, there are various MMU features that are more important than the rest of the arch.
03:57:07 <nyc`> zhiayang: One case is big gaps in page sizes. Another is a broad spectrum of page sizes. Another is tiny pages like VAX (512B). Another is PAE where physical is much larger than virtual and the number of physical pages makes per- physical page overhead a huge issue with limits on kernel virtualspace.
03:58:21 <nyc`> zhiayang: Demonstrating that the algorithms handle all those cases needs to take on all those arches to do.
04:00:59 <nyc`> 32-bit MIPS looks like the most important one: that has 1KB pages,.a broad spectrum of page sizes, sharply limited kernel virtualspace, and physical greater than virtual (e.g. PAE).
04:04:50 <nyc`> x86-32 PAE combines physical greater than virtual with a large gap in page sizes, so it's meaningful, too.
04:08:42 <nyc`> The 64-bit arches don't really demonstrate much besides the performance boost from operating without pressure on virtualspace, but they're considered more important ir relevant than 32-bit for actual use rather than showing how well the algorithms can work under hostile conditions.
04:10:49 <nyc`> I suppose the difference from most kernels on 64-bit would be better use of large pages.
04:12:51 <zid> I'm waiting for GINOURMOUS PAGES
04:13:21 <nyc`> zid: They already exist on various arches.
04:13:38 <nyc`> SPARC has 16GB pages, for example.
04:14:02 <zid> those are still HUGE imo
04:14:30 <nyc`> It has powers of 8 starting from 8KB.
04:14:32 <zhiayang> you should be able to set a bit on the address in cr3 to get 256TiB pages
04:14:51 <zid> yea I was thinking 1TB+ for GINOURMOUS
04:15:21 <zid> Just use it for an identity mapped long mode in a single tlb entry
04:15:28 <nyc`> zid: The spectrum matters more than just the largest.
04:15:41 <zid> the spectrum honestly doesn't matter
04:15:55 <zid> it's harder to cache and takes processing time shuffling them around
04:16:01 <zid> you just want a single compromise size, and a large size
04:16:12 <nyc`> No, the spectrum makes all the difference.
04:16:19 <zhiayang> at this point i'm not sure if i want to support large pages
04:16:26 <zid> Try designing an efficient TLB with more than a few sizes
04:16:30 <zhiayang> let alone huge or ginormous
04:16:38 <zid> Intel decided they couldn't, or it wasn't worth it, afterall
04:16:44 <catern> zhiayang: isn't 256TiB the entire supported virtual address space
04:16:51 <zhiayang> catern: yes, that's the joke
04:16:53 <catern> oh
04:16:59 <nyc`> The point is to increase the TLB reach.
04:17:00 <catern> it was funny then
04:17:04 <zhiayang> thank you
04:17:26 <zid> Don't forget about PML5!
04:17:55 <zhiayang> are they gonna give up making names for those
04:18:16 <zid> They already did with pml4 lol
04:18:19 <zhiayang> i want my pdptpt and my pdptptpt
04:18:31 <zid> page library page directory page table entry
04:18:35 <zid> page bookshelf page library page directory page table entry
04:18:50 <zhiayang> hm shouldn't bookshelf be smaller than library
04:19:01 <zhiayang> but what if we want a pml6 what comes after library???
04:19:08 <zid> page library(as in building) page bookshelf page library(as in collection) page directory page table entry
04:19:10 <zid> fixed
04:19:26 <zhiayang> PLAIB is a good acronym
04:19:28 <zid> and the difference in all the docs will be PL and Pl
04:19:33 <zhiayang> HAHA
04:19:46 <nyc`> Just get rid of the radix tree and have a fully associative TLB with lots of page sizes and ASN's and such.
04:19:58 <zid> now implement it in hardware
04:20:12 <zid> don't forget straddles and multiple maps etc
04:21:15 <nyc`> zid: What I'm proposing is a lot easier to implement in hardware than the radix tree walker and atomic updater by a long shot.
04:22:30 <nyc`> I also have special sauce that'll make better use of larger pages.
04:23:07 <nyc`> (original algorithms, unpublished, so I am unwilling to disclose)
04:23:31 <zid> ohh so you're a schizophrenic
04:24:23 <dormito> anyone know how to query what the default output type (elf,coff,binary) of ld is?
04:24:24 <nyc`> No, I have one original algorithm I could publish a paper on if I had backing from a uni or a company.
04:26:51 <nyc`> zid: I actually did an IBM-internal whitepaper in 2003 about it, but I didn't get the team I needed for the Linux retrofit, so I did other things and odds are it was never understood and quickly forgotten.
04:28:45 <nyc`> zid: It's very invasive, so it's in some ways easier to write a kernel from scratch than to try to merge against the onslaught of upstream changes.
04:31:21 <nyc`> zid: Once I get to where I can fully defend my priority, I'll obviously describe things. Obviously I'm going to post the code and probably try to publish a paper even without a big sponsor.
04:32:53 <nyc`> Most of the contiguity opportunities are not enormous pages, though. The limitation is really the availability of contiguous memory.
04:44:20 <nyc`> zid: VAX and MIPS with a 1KB granule are there to show how much better my algorithms can line up pages in a row as well as keep overhead light, but really, it's not magic or anything. It's some marginal improvement and moving around between various different kinds of trade-offs. I'm relatively certain no one has ever done it in any operating system before. There's no delusion of grandeur.
04:45:20 <catern> look at that wall of text, definitely schizophrenic ;)
04:45:42 <zid> He kind of proved it to me after the accusation :P
04:46:02 <catern> we should get rid of virtual memory entirely
04:46:23 <zid> Move it into the dram controllers ;)
04:46:38 <nyc`> Seymour Cray had a good quote about virtual memory.
04:46:40 <catern> nice
04:47:12 <nyc`> Memory is like an orgasm: it's better when you don't have to fake it.
04:47:24 <zhiayang> why tf did amd design sysret to be so inane
04:47:48 <zhiayang> "If SYSRET is returning to 64-bit mode, the CS selector is set to this field + 16. SS.Sel is set to this field + 8, regardless of the target mode."
04:47:49 <zid> x86 has always had issues with syscalls in general
04:48:07 <zid> despite having had like 10 entry methods, they've all sort of blown chunks
04:49:11 <zhiayang> like the typical setup is 0: null, 0x8: r0 code, 0x10: r0 data, 0x18: r3 code, 0x20: r3 data
04:49:22 <zhiayang> and the syscall semantics seem to support this setup
04:49:27 <zhiayang> but sysret??
04:49:29 <zhiayang> jee
04:49:29 <zhiayang> z
04:50:28 <zhiayang> so i need a duplicate ring3 code at offset 0x28...
04:50:52 <nyc`> x86 is terrible at context switching. It should be retired so RISC can take over.
04:53:00 <nyc`> Context switching should be swift, painless, and lightweight. It's better when CPU's have less state to save etc.
05:08:51 <isaacwoods> zhiayang: it's so you can have "Ring-3 32-bit code segment", "Ring-3 data segment", "Ring3 64-bit code segment"
05:09:03 <isaacwoods> look what happens when you return to compat mode
05:09:58 <zhiayang> i mean they could've done it so it was 64-r3-cs, r3-ds, 32-bit-r3-cs
05:10:22 <zhiayang> i don't really care about compat mode
05:15:24 <isaacwoods> zhiayang: neither, I just put a null segment there
05:16:02 <zid> enforcing the ordering to have things go smoothly makes sense in concept, slightly grotty in practice ofc
05:16:08 <zid> but just re-order it how it wants it and be done
05:17:20 <zhiayang> yea, it's just a bit annoying is all
05:55:25 <mischief> ~
05:55:28 <mischief> *cough*
05:55:31 <zhiayang> \o
06:31:03 * geist yawns
07:16:53 <froggey> that was a "fun" bug. I rewrote a chunk of my VM and ended up breaking the framebuffer
07:17:02 <zid> Correctly played
07:17:17 <zid> Fixng bugs and having things work again afterwards means you forgot something really insidious
07:17:40 <zid> means you probably accidentally created a root exploit. If it all explodes it means you defensively made it explode and are a genius.
07:18:24 <froggey> ah, but the bug turned out to be really insidious and somewhere effectively unrelated to the framebuffer at all
07:18:44 <zid> that's perfect, the problem is when it doesn't break anything
07:18:47 <zid> then you're in real trouble
07:20:36 <froggey> the VM changes broke the acpi driver because it was using an obsolete syscall, and the acpi driver crashed halfway through probing the size of the framebuffer's pci bar
07:20:58 <knebulae> @froggey: what are these "drivers" you speak of?
07:21:17 <froggey> and that left the video device in a completely broken state
07:21:31 <froggey> software that attempts to make hardware do something useful
07:21:53 <froggey> but usually just cause trouble
07:23:02 <zid> You'd have found it sooner if the obsolete syscall brought down your entire system though
07:23:05 <zid> So you failed on that respect
07:23:33 <froggey> so I spent my weekend chasing ghosts in the VM system, when all I had to do was fix the obvious & easy crash
07:23:47 <froggey> I know, that's actually on my todo list :(
07:24:21 <zid> I wish my stuff was crashing
07:24:26 <zid> I just get really weird behavior
07:38:10 <geist> this is why i usually add lots of asserts and internal consistency checks when writing new code
07:38:27 <geist> it usually really pays off in the end since you can catch a lot of really bad stuff before it goes heisenbug
07:38:42 <geist> it looks excessive, but i think it really really pays off
07:43:58 <knebulae> How do you guys go about testing? I mean, with normal application software, you can write unit tests, etc., but with kernel code its a big different. How do you make sure to validate that your kernel does what it says under the right conditions?
07:44:20 <geist> where you can unit test you should try to, but in general i do lots of stress testing
07:44:24 <knebulae> I guess I'm asking more about automated testing btw.
07:44:35 <geist> ie, write test cases that really pushes the kernel and tries to make it fall over
07:44:41 <geist> for stuff like schedulers, locks, etc
07:44:51 <knebulae> @geist: ok
07:45:24 <geist> that plus a lot of asserts typically means the system will trip an assert if something tets out of whack
07:45:25 <zid> geist: what assert do I add to die when IRQ11 fires 3 seconds after I cause it externally
07:45:40 <knebulae> @geist: I just don't know that my tests would stress harder than say building a "big" application (like glibc) and putting it through its paces.
07:45:56 <knebulae> or a rdbms
07:46:01 <geist> zid: oh easy, you assert that the thing you're supposed to signal when the irq doesn't fire took too long
07:46:13 <geist> but.... trying to write test cases agaiinst real hardware is tricky
07:46:19 <geist> since real hardware can misbehave
07:46:25 <knebulae> right
07:46:38 <geist> but in that case, for exmaple, you could wait on an event or object with a timeout, where the timeout is some large number like 10 seconds
07:46:46 <zid> geist: It's bochs misbehaving, as far as I can tell, I can't get qemu to work to test it
07:46:50 <geist> then put the device throgh it's paces, and if it misses an irq it'll hit the timeout
07:47:00 <geist> missing an irq is one of the easier ones to assert for
07:47:02 <zid> I ping bochs, the green packet light lights up, then 3 seconds later my kernel prints "irq11 just happened"
07:47:14 * geist nods
07:47:24 <geist> that's difficult, but in that case you're testing it, it just involves a human
07:47:41 <zid> yea I'd love to see the assert that catches why that happens though :P
07:47:47 <zid> The diagnostically relevent if() for that
07:47:54 <geist> knebulae: sure, you can write an app that creates say a thousand threas, runs them in parallel, verifies that they each get some amount of time slice (so they're not starving)
07:48:06 <geist> you can create piles of threads that contend on a mutex, verify that the mutex is functioning
07:48:09 <geist> etc etc
07:48:16 <knebulae> very good.
07:48:35 <geist> i always write a fun fibonacci test: create a process that uses either threads or more processes to recursively compute fibonacci
07:48:45 <knebulae> taking your time and doing it right is very time consuming.
07:48:54 <geist> ie, fibo() { create 2 threads; fibo(n-1); fibo(n-2); }
07:49:03 <geist> ends up creating a shitton of threas and/or processes
07:49:07 <knebulae> @geist: I like that idea
07:49:10 <geist> knebulae: nothing is free
07:49:30 <knebulae> @geist: it's not work if you love it.
07:49:41 <geist> my experience is having an unstable system will waste *far* more time in the long run than any amount of tests you write
07:49:47 <froggey> that's cute. I'm gonna try that now
07:50:18 <geist> https://github.com/littlekernel/lk/blob/master/app/tests/fibo.c#L33 is the one for LK
07:50:25 <eryjus> knebulae: i do write a lot of output to the serial port and I can grep again error conditions if i form my output strings well. With the right cable, you should be able to do that on real hardware as well (i do with the rpi)
07:50:41 <geist> wrote this one like 15 years ago for newos: https://github.com/travisg/newos/blob/master/apps/fibo/main.c
07:50:44 <zid> geist: You just failed all my push hooks, mixed declarations and code galore
07:51:04 <geist> oh for fucks sake can it
07:51:09 <knebulae> @eryjus: I am not to the point that I want to use hardware other than the framebuffer yet.
07:51:25 <geist> i do not give a shit about whatever code style you may have, dont start
07:51:36 <zid> calm down
07:51:37 <zid> christ
07:51:53 <geist> okay to stop: *huge* pet peeve... i post some thing i put a lot of effort and time into, and someone jokingly comments about some trivial detail
07:51:59 <geist> that is a button for me, for sure
07:52:10 <geist> sorry, not your fault, just had to vent
07:52:12 <knebulae> It's par for the course around here.
07:52:18 <geist> i know, one of the reasons i dont like to do it
07:52:27 <geist> nor do i ever generate disk images or screenshots for anything i do
07:52:39 <geist> because on the average people will sit around and snark at shit instead of actually looking at it
07:52:41 <zid> geist: You'd be better taking it as a compliment that that's the thing I found to complain about
07:52:53 <geist> zid: ya i know. i'm not mad at ya. it's just a button of mine
07:52:57 <zid> "If that's the best you got I'm doing well"
07:53:14 <geist> the biggest case of it was years ago i spent like 100 hours doing this little 6809 machine, doing the hardware layout, printing the PCB, etc
07:53:31 <geist> i was sooo proud of it, posted it somewhere and fucking first comment was some dumb shit like 'nice readme, dude'
07:53:37 <geist> because the README.md was basically empty
07:53:57 <zid> I only care about people who I care about
07:54:17 <zid> So if my engineer friends had concerns, I'd hve concerns, if some fuckwit 12 year old does, I don't
07:54:34 <geist> like many people i know, i have a fairly strong impostor syndrome
07:54:46 <geist> one of the ways to deal weith it is to overcorrect by really really trying to do a fantastic job on something
07:54:53 <geist> and really go above and beyond. it has driven me for years
07:55:03 <eryjus> zid: s/care about/respect/
07:55:06 <zid> I have good engineering arguments for not mixing declarations and code, but you might melt :P
07:55:14 <knebulae> @geist: dude you were on here in 2003 or 2005. It's been a decade. You're not an imposter anymore man...
07:55:15 <zid> eryjus: You fall into the latter category there.
07:55:33 <geist> so as a result i'm also really brittle when it comes to criticism
07:55:33 <zid> I don't care about your correction because I don't know who you are enough to care about your opinion
07:55:44 <geist> and *giving* criticism. i feel like i dont have the right to criticize
07:55:47 <eryjus> thank you -- not sure I've earned that but I appreaciate it
07:56:01 <geist> knebulae: sure, but see that stuff isn't *rational*
07:56:25 <geist> its irrational. in some sense the more praise folks heap on you makes it worse, because then the dichotomy between what you experience and what you feel is worse
07:56:34 <zid> geist "touchy about things sometimes" brecht
07:56:43 <zid> Better than my subtitle :P
07:56:47 <zid> "total asshole"
07:56:48 <geist> mostly that one. that's a mega button for me
07:57:05 <geist> plus you wouldn't believe how much people will nit about code style at work
07:57:18 <geist> my personal projects are like my space where i can do what i want and not feel like i have to deal with that
07:57:27 <knebulae> @geist: shit, a decade and a half!
07:57:35 <zid> geist: wanna fix my bochs? :P
07:57:36 <geist> plus stuff like newos was written 15-20 years ago
07:57:43 <zid> If you can do it I'll be impressed
07:57:54 <geist> knebulae: here? yah i showed up in mid 2001
07:58:05 <geist> only one i know that's still around from before that was air
07:58:07 <knebulae> @geist: I know ;)
07:58:21 <knebulae> lol, well, sorta.
07:58:35 <geist> i was a young snot back then. newos had just been slashdotted (which was still a big deal bcak then)
07:58:39 <geist> so i was pretty cocky
07:59:25 <geist> every so often i get nostalgic and try to modernize newos. i have some branch somewhere where i ran it through some code formatters and started working on getting it working again with modern gcc
08:00:03 <zid> Failing that, you can help me get IFCS working, it always gives me 0s for some reason
08:00:28 <geist> hmm, what is IFCS?
08:00:34 <zid> hardware mac checksumming
08:00:49 <geist> ah
08:00:51 <zid> it's supposed to just be an enable bit, and a start/offset pair
08:01:13 <zid> I could manually calculate it but it's annoying that the hw checksumming doesn't work :P
08:01:23 <geist> what context is this?
08:01:28 <zid> e1000 driver
08:01:39 <geist> ah. against an emulator?
08:01:52 <zid> yea bochs, qemu uses it too but I can't for the life of me get networking working in qemu :P
08:02:12 <geist> oh. i'd recommend just digging into qemu's side of the driver and add some traces
08:02:21 <geist> that i usually find gets you right to the point
08:02:23 <zid> yea but I can't get qemu networking working
08:02:27 <zid> I could do the same to bochs though
08:02:31 <zid> the source is pretty good
08:02:31 <geist> well, then get that working first
08:02:42 <zid> I tried for like an hour this morning and made precisely 0 progress
08:02:51 <zid> except having to restart dhcpd on eth0 10 times
08:03:04 <geist> what about user netowkring in qemu? it usually works pretty nicely
08:03:11 <geist> doesn't require any external anything
08:03:28 <zid> I can't ping it unless I use tun/tap style networking, and the thing I am testing with is ping
08:03:45 <geist> sure, but if you just want to get the driver working, start by not pinging it
08:03:52 <geist> and just get the dhcp stuff working
08:04:11 <geist> you can punch tunnels in the user networking though, though probalby not ping per se
08:04:19 <geist> but you can try to send it a udp packet or tcp packet though
08:04:20 <zid> well I don't have any dhcp code, and I don't know the protocols, etc
08:04:29 <geist> for the purposes of making sure the nic is working that should suffice
08:04:41 <geist> send it some pre canned udp packets, and or reply with the same
08:04:43 <geist> that's pretty easy
08:05:51 <zid> I think I will just put up with the weird lag in bochs until I figure out why ifcs doesn't work (which I am assuming is the reason ping isn't working, for now)
08:05:59 <geist> okey dokey.
08:06:09 <geist> i can only suggest ways to help you solve your problem
08:06:17 <zid> yea but it was more "change the problem space"
08:06:45 <catern> is there any channel like #osdev but for userspace stuff? like writing C libraries, shells, RPC systems, routers, that kind of stuff
08:07:02 <geist> no, i'm trying to help you debug your driver by changing the way you test it
08:07:10 <geist> until you get it working, then you can switch back to pinging it or something
08:07:56 <zid> https://cdn.discordapp.com/attachments/417023075348119556/544610534276857876/unknown.png
08:08:19 <zid> top = wireshark, bottom = pings coming back as failures, fun fun
08:08:26 * geist nods
08:08:56 <zid> imma quickly try tacking on a manually calculated fcs
08:10:52 <zid> oh it's actually really complicated, heh
08:55:17 <Telyra> zid: Generally if I'm testing networking stuff (which is 90% of my project) I use VirtualBox as the hypervisor. The network bridging is a lot saner to configure and since it's a proper hypervisor you don't have to deal with the nightmare that is how goddamn slow Bochs is
08:57:38 <zid> Telyra: Can vbox do e1000?
08:57:44 <Telyra> Yep.
08:57:48 <Telyra> A few varieties of it.
08:58:32 <zid> ah neat I'll try vbox
09:12:34 <zhiayang> is there any way to make bochs go faster
09:12:46 <zhiayang> it does not appear to be using my cpu much, so what is it constrained by
09:13:03 <Telyra> Not really. It's a CPU emulator, so it's doing everything in software
09:13:31 <Telyra> One CPU core in Bochs is going to be running at whatever it can on one physical host CPU core
09:13:47 <zhiayang> it does not appear to be doing “whatever it can” though
09:13:56 <zhiayang> it’s painfully laughably slow
09:14:21 <Telyra> I genuinely don't use Bochs for anything because of how uselessly slow it is
09:14:34 <zhiayang> 60 seconds to get to the efi shell, and i thought qemu was bad
09:14:37 <Telyra> Even when tuned properly it's still an order of magnitude slower than useful
09:14:49 <zhiayang> how does one tune it?
09:15:16 <zhiayang> i’m not planning to use it as my primary, but it can inspect a lot more cpu state than qemu
09:15:18 <Telyra> Used to be you'd fiddle with the ips= setting in your bochsrc until you got to where you wanted it
09:15:35 <Telyra> But I haven't used Bochs since the early 2000s
09:15:45 <zhiayang> is the idea not to set that to an absurdly big number?
09:17:57 <Telyra> I think you're supposed to compile bochs with --enable-show-ips to benchmark your machine
09:18:06 <Telyra> then use that to determine what ips setting to use
09:18:14 <zhiayang> hmmmm
09:18:16 <Telyra> Basically, uh
09:18:18 <Telyra> Don't use bochs
09:18:29 <Telyra> Or don't use bochs with any expectation to have any sort of sane timing
09:18:43 <Telyra> It's deterministic, and that's about what it's good for
09:19:29 <knebulae> I had to ditch bochs due to lack of uefi support (although some people with much more time on their hands than I have made it work).
09:19:45 <zhiayang> knebulae: it works
09:20:04 <zhiayang> you just change some pci settings and point at at the ovmf firmware
09:20:07 <Telyra> Reminder that this is the emulator that still has you input CHS values for hard disk images in 2019
09:20:14 <knebulae> I have qemu, vbox and real hardware. Bochs was great back in the day; now? meh.
09:20:17 <zhiayang> ya it’s fucked up
09:20:46 <zhiayang> i figured c200 h16 s64 gives me exactly the number of bytes as my disk image
09:20:56 <knebulae> @zhiayang: right.
09:20:56 <zhiayang> but nooooo sectors per track cannot exceed 63
09:21:44 <knebulae> @zhiayang: off by one in your calcs? or do disks start at sector 1. Honest question, I haven't touched a driver yet. :p
09:22:23 <zhiayang> it’s just geometry, c * h * s * 512 should be the bytes of the disk image
09:22:29 <knebulae> ahhh
09:22:44 <zhiayang> i’m not foolish enough to write anything using chs
09:22:51 <knebulae> @zhiayang: so def. no zeros.. :)
09:23:08 <zhiayang> anyway, wrt efi: i can load my kernel, exit boot services and jump to it
09:23:20 <zhiayang> so for all intents and purposes uefi appears to work on bochs
09:24:00 <knebulae> @zhiayang: mental note taken.
09:24:11 <zhiayang> but there might end up being some hardware that’s misconfiged, who knows
09:24:32 <froggey> so many threads...
09:24:43 <zhiayang> hm?
09:25:35 <froggey> Í'm trying geist's fibonacci test. it spawns a whole bunch of threads
09:26:09 <zhiayang> ahh yes i saw that earlier
09:26:13 <zhiayang> i should try that
09:43:51 <knebulae> is anyone using ada?
09:44:40 <knebulae> I went on a kick about a year ago, and I noticed the noise around ada is kicking back up a little with their Spark/Ada2020 push. I was really just curious to see if anyone is doing a kernel, and if so, how the performance was compared to C.
09:49:08 <knebulae> I concluded that it was impossible to do a kernel in Spark though.
09:52:02 <froggey> "fib(10) = 55. took 2,331.8s" I think I have some scaling problems
10:05:57 <geist> froggey: yah it also changes depending on if you start the threads then wait on them all at once vs start one, wait, start another, wait
10:06:06 <geist> the former is what you want, it really lets it grow
10:27:16 <nyc`> I slept through my doctor's appointment. It'll be time to attack the sparc64 boot issue when I finish sleeping. Worst case I just statt working on ARM instead, though theoretically one should be able to fire up a Linux guest or do qemu sparc32(!) emulation to render iso9660 disk images bootable. Best case I find firmware that can tftpboot or a way to get -kernel working.
10:44:10 <mobile_c_> klange: in "while (table->d_tag) {" what would happen if that is 0
10:47:34 <mobile_c_> wtf i have that aswell in my linker o.o
11:16:24 <mobile_c_> god it difficult to convert my string reading functions back into freading functions
11:16:52 <mobile_c_> cus for my PT_DYNAMIC in MY linker i have
11:16:54 <mobile_c_> read_fast_verify(library[library_index].array, library[library_index].len, &library[library_index].tmp99D, (library[library_index]._elf_program_header[i].p_memsz + library[library_index]._elf_program_header[i].p_offset));
11:16:55 <mobile_c_> __lseek_string__(&library[library_index].tmp99D, library[library_index]._elf_program_header[i].p_memsz, library[library_index]._elf_program_header[i].p_offset);
11:17:20 <mobile_c_> if (library[library_index].PT_DYNAMIC_ != 0) { library[library_index].dynamic = library[library_index].tmp99D; ... }
11:20:58 <mobile_c_> wait no
11:49:15 <mawk> [ 0.713888] overflow: 0000 [#1] SMP PTI
11:49:17 <mawk> what's that
11:49:35 <mawk> is it what happens if I provide no ACPI, no SMBIOS, no MP, no whatever ?
11:49:48 <mawk> that happened when I tried to send irq number 4 for the first serial port
11:50:09 <mawk> when I do it fast enough after the boot, anyway
11:50:34 <mawk> if I do it later I have VM exit number 0x80000021, which means "hardware error" and "invalid guest state on entry"
11:50:57 <mawk> that's a mystery