channel logs for 2004 - 2010 are archived at http://tunes.org/~nef/logs/old/ ·· can't be searched
#osdev2 = #osdev @ Libera from 23may2021 to present
#osdev @ OPN/FreeNode from 3apr2001 to 23may2021
all other channels are on OPN/FreeNode from 2004 to present
http://bespin.org/~qz/search/?view=1&c=osdev2&y=22&m=1&d=16
00:49:00 <Clockface> ok, so this is pretty cool
00:50:00 <Clockface> i figured out that the somewhat unusual/braindead way of doing things mean i can write mine to run in the userspace of a real operating system if i swap out the bootloader and drivers
00:51:00 <Clockface> well, part of it at least
00:51:00 <Clockface> its enough that it makes my life a hell of a lot easier
09:33:00 <kazinsal> I just wrote a 3500 word forum post on the history and insanity of x86 memory management because a bunch of folks liked my (much shorter) effortposts about the VAX and I feel intellectually beat after explaining the history of Intel's "just pile it on, it'll be fine" architecture
09:38:00 <geist> oh wow
09:50:00 <kazinsal> people were like "man that's more effort than we were expecting about VAX, what else you got"
09:50:00 <kazinsal> their mistake
10:22:00 <geist> kazinsal: is there a link to this discussion?
10:23:00 <kazinsal> depends on if you have a SomethingAwful forums account or not, pretty sure YOSPOS is behind the paywall
10:23:00 <geist> ah oh well
10:24:00 <kazinsal> it started because someone posted a thread called "tell me about the DEC Alpha" and then someone asked VAX and I chimed in
10:24:00 <geist> i never 100% got clear on precisely how the MMU worked on the alpha
10:25:00 <geist> ie, is it a pure software TLB + software routines implemented in PALcode?
10:25:00 <geist> (i think thats the answer)
10:25:00 <kazinsal> so then the thread got renamed to "tell me about DEC hardware" and I joked about writing something on the x86's bizarre memory management and a few people were like "oh god please do"
10:25:00 <geist> and the standard PALcode implements basically a standard page table
10:25:00 <froggey> geist: from the little I poked around with Alpha, that's exactly it
10:26:00 <geist> yah. i wonder if the 2 or 3 standard palcode libraries implemented it the smae way?
10:26:00 <geist> i understand there was a DEC one for VMS and unix and then there was the one for NT
10:26:00 <kazinsal> probably. I think raymond chen wrote a bunch of stuff about NT on Alpha, let me dig that up
10:26:00 <froggey> dunno. I know that the source for at least one palcode is available
10:26:00 <geist> wouldn't be surprised in the NT one was nerfed to be 32bit (since NT ran in 32bit mode) and thus used 4 byte PTEs etc
10:29:00 <geist> froggey: yah i guess it was set up such that you can configure the TLB miss exception to directly vector to PAL code
10:30:00 <geist> or maybe regular exceptions are always run after at least one instruction of palcode
10:30:00 <geist> and the whole exception delivery mechanism is itself implemented in palcode
10:30:00 <geist> has a certain 'microcode all the way down' feel to it
10:31:00 <froggey> yeah, the cpu vectors directly to a fixed palcode entry and the palcode is what signals any kind of "regular" exception
10:32:00 <kazinsal> "An Alpha AXP program running on 32-bit Windows NT still has full access to the 64-bit registers and can use them to perform 64-bit computations. It could even use the full 64-bit address space, if you were willing to jump through some hoops."
10:32:00 <kazinsal> neat
10:32:00 <froggey> iirc all exceptions & interrupts went to palcode and it was responsible for implementing regular exceptions
10:33:00 <geist> i haven't looked into qemu alpha support but i wonder if qemu attempts to run palcode, or if it emulates it as if it already had palcode loaded...
10:34:00 <froggey> fun fact: the last model of lisp machine sold by symbolics was an alpha axp workstation running an Ivory emulator
10:34:00 <froggey> running on stock hardware ended up being cheaper/faster than designing custom hardware
10:36:00 <geist> neat
10:36:00 <geist> the one time i ever used an alpha workstation it was a screamer
10:36:00 <geist> i had a little DEC Multia years later, after it was obsolete
10:37:00 <geist> alpha hardware is still hyper expensive on ebay, must be a real demand for them in production
10:51:00 <nomagno> geist: I gave up and used a FLOSS alpha VM made by an angry ex-employee, it was buggy but good enough to do the OpenVMS testing I needed
10:52:00 <nomagno> OpemVMS is getting an x86 port, apparently
12:47:00 <geist> yah i think it's in some sort of beta and/or released now
15:44:00 <Griwes_> geist: what's the main point behind the hierarchical design for VMARs in zircon? I'm having trouble imagining a situation where I'd want to create a child VMAR and lower its permissions, since for the cases that come to mind, the creation of it would be pretty much right next to the call to zx_vmar_map. Is there some cases of foreign processes wanting to map something into your address space and needing (from the non-foreign perspective) to have lowe
15:44:00 <Griwes_> permissions that I'm missing?
17:01:00 <nomagno> Is there a generic term for a program that is both an assembler and a bytecode interpreter?
17:20:00 <sortie> interpreter?
20:46:00 <Ermine> Translator?
20:48:00 <nomagno> I ended up splitting the interface into 'assembler' and 'exec'
21:39:00 <geist> Griwes: whether or not you lower the permissions, each VMAR (and sub vmar) has a handle (or no handle, you can throw away the last copy of it) and thus only users with the handle can operate
21:39:00 <geist> so for example you can carve off the frst 4GB of the address space and give it to the loader, and then no other code has that handle and thus cannot manipulate anything in the 0-4GB range
21:40:00 <geist> or you can create a sub vmar + handle and give it to another process and they can then only manipulate that range of the address space
21:40:00 <geist> permission wise yes the heirarchy of permissions is somewhat less used, also since the handle itself has a right which can additionally layer on top of it
21:40:00 <geist> but you can for example create a range of your address space -X permission
21:41:00 <geist> and then give the handle to anyone. they might create another sub vmar, but it would break the permission model if they could add +X to their child
21:41:00 <geist> since it would defeat the purpose of having -X in the first place
21:42:00 <Griwes> the first example you gave has nothing to do with *security*, right? because you must assume that whoever is in your address space can find that handle by probing memory?
21:42:00 <geist> no. they cannot period
21:42:00 <geist> the handle is private to the aspace, cannot be cloned externally, etc
21:42:00 <Griwes> right, but zircon has an in-process loader, right?
21:42:00 <geist> you cannot find a handle to a thing by probing. the only way to do it would be to have code in that process actually duplicate it
21:43:00 <Griwes> so non-loader code could find that handle in memory?
21:43:00 <geist> the handle doesn't exist in memory. you'd have to probe around the actually find it in the .data segment of the loader
21:43:00 <geist> however there's also a situation where the loader actually throws the handle away
21:43:00 <geist> one of the differences in VMARs to most zircon objects is throwing away the last handle to the vmar does not destroy t
21:43:00 <geist> it stays in place and then that part of the aspace is basically locked down
21:44:00 <Griwes> yeah I've read the docs ;p
21:44:00 <geist> so you can also create a locked down procss by either externally loading it (and thus there is no local handle to the vmar where code is) or loading, and then discarding it
21:44:00 <geist> but yes, you could sniff around and try to find the handle in the loaders data segment i think in the normal case where the handle is left around
21:45:00 <geist> there is somewhat of a combination of external and in-process loaders for normal processes on fuchsia, so yes there is some general need for fiddling with the local aspace after it starts, for dyload() etc
21:45:00 <geist> or at least i think that's what goes on
21:46:00 <Griwes> heh
21:46:00 <Griwes> alright, thanks for the commentary
21:46:00 <geist> but yeah re: heirarchial permissions, it's mostly so you can't just create a sub vmar which raises the elevation, otherwise it defeats the whole purpose of having permissions on it
21:46:00 <Griwes> I'm still trying to square kernel vs userspace vmm in my head
21:46:00 <Griwes> currently I'm at the stage where I hate both
21:46:00 <Griwes> :D
21:47:00 <geist> on zircon or in general? the heirarchial VMAR stuff is AFAIK unique to zircon
21:47:00 <Griwes> (well, "hate")
21:47:00 <geist> most systems are flatter than that
21:47:00 <Griwes> for my kernel
21:47:00 <geist> when you say kernel vs usespace you mean implementing a VMM in onr or the other, or something like 'kernel responsibilities' vs 'userspace responsibilities'?
21:47:00 <Griwes> yeah I'm thinking flat but I'm trying to figure things out and there appears to be no approach that I actually, actively, like
21:48:00 <Griwes> implementing a VMM
21:48:00 <Griwes> was it L4 that had a fully userspace one?
21:48:00 <geist> ah i once implemented a kernel where the VMM was implemented in user space. was interesting, but didn't get far enough to really explore what it was going to look like
21:49:00 <geist> in that case i still had the PMM implemented in kernel space. so it had some specialized syscalls the vmm would use: pmm_get_pages, pmm_put_pages and a bunch of page table calls: mmu_map_page, mmu_unmap_page, mmu_set_protection, etc
21:49:00 <geist> had a way to register for page faults to be delivered into the VMM process, and then took it from there
21:50:00 <geist> having the pmm in the kernel meant the kernel could still internally grab pages for it own use, but most of the list f free pages were consumed and returned by the userspace VMM
21:50:00 <geist> then that meant all the usual VMM calls were IPC to the VMM process
21:50:00 <geist> i think it would have worked okay
21:51:00 <tusko> does anything about this syscall and test seem clearly incorrect to you, according to std. practices? https://bpa.st/VEXA
21:51:00 <bslsk05> bpa.st: View paste VEXA
21:51:00 <geist> oh also there were some syscalls for mmu_create_aspace -> handle to aspace, mmu_delete_aspace, etc
21:51:00 <geist> and as processes were created in user space they'd call into the VMM to allocate a fresh handle to associate with a process
21:51:00 <geist> something like
21:52:00 <geist> create_process -> proc_handle, create_aspace -> aspacE_handle, bind_apsace_to_process(process_h, aspace_h) as part of one of the steps of process creation
21:52:00 <geist> (this was a µkernel)
21:52:00 <geist> for zircon sicne the IPC is asynchronous we decided to just leave the VMM in the kernel as well
21:53:00 <Griwes> yeah, that makes sense
21:53:00 <geist> the µkernel i was talkig abot where VMM was in user space was fully synchrnonous, so the kernel only needed to allocate memory for its own purposes when you created/freed objects, or mapped pages (for page tables)
21:54:00 <geist> otherwise it didn't really consume anything from the pmm
21:54:00 <Griwes> user-space vmm has silly problems like "how does the syscalls mechanism verify the vdso locations" or "how do I manage permissions for the mmu syscalls"
21:55:00 <Griwes> and I've been spinning on trying to make a decision for quite some time now
21:55:00 <j`ey> heh, linux 3.9 defconfig took 6mins to build, my minimal config for 5.16 takes 30mins :/
21:55:00 <Griwes> (like, before I build something non-trivial in the kernel or whatnot)
21:55:00 <geist> Griwes: well simplest one for the VMM permission is to simply have some priviledged 'i'm a superserver' bit set on the process and move on
21:56:00 <Griwes> yes
21:56:00 <geist> the more capability based one would be to have some sort of priviledged handle to the PMM/MMU kernel feature that the VMM has to pass on every syscall
21:56:00 <Griwes> but that raises further questions of "can I have more than one of these?"
21:56:00 <geist> we cheated a bit on zircon by having a 'resource' handle that unlocks high end permissions for things like that
21:56:00 <Griwes> "can different processes select different policies by talking to different vmms?"
21:56:00 <geist> sure. that'd be easy since you could creat ean aspace object
21:57:00 <geist> how about create_aspace -> aspace_handle
21:57:00 <geist> bind_aspace_to_vmm(aspace_handle, handle to page fault IPC/etc)
21:57:00 <Griwes> right; I'm not saying it's impossible, it's just that it raises so many questions that I'd need to find at least draft answers to to be comfortable with going that way
21:58:00 <geist> now every time you create an apsace you bind it such that all future page faults, etc get delivered to the IPC/etc channel the VMM has
21:58:00 <Griwes> I like to understand the consequences of choices I'm making and the decision tree here is so damn deep
21:58:00 <geist> that way if some other thing wanted to take on that responsibility the kernel doesn't care
21:58:00 <geist> i think the key is always think in a capability like way: to create a Thing you must have a handle to Another Thing
21:58:00 <geist> and then you can have operations that glue two or more things together, provided you have handles to it
21:59:00 <geist> obviously there's bootstrapping issues, and the first process probably cheats and gets handles to Uber Things delivered to it by the kernel, but then user space chooses how to subdivide responsibility
21:59:00 <Griwes> right
21:59:00 <Griwes> but the first process needs handles delivered to it anyway
21:59:00 <Griwes> so at least that part is not a big deal
21:59:00 <geist> in the aspace handle case you now have a capability to lock off the mmu_map/unmap calls
21:59:00 <geist> mmu_map_page(aspace_handle, ...)
22:00:00 <geist> the only real magic capability yo have there is the ability to create the aspace in the first place.
22:01:00 <Griwes> right; that's probably managed by the same capability that allows creating new processes
22:01:00 <geist> also if you keep aspace handles and process handles separate you can do some fun stuff like have two processes use the same aspace
22:01:00 <Griwes> or at least something of a similar magic level
22:01:00 <geist> or have aspaces with things mapped that no process is using
22:01:00 <Griwes> I really really don't want to have two processes use the same vas
22:01:00 <geist> or some way to create a prototype aspace that maybe you add some sort of aspace_clone() call
22:01:00 <Griwes> but yeah having a detached vas seems reasonable
22:01:00 <geist> or maybe that's exactly the way you create new aspaces: clone an existing, empty one
22:02:00 <geist> and thus you have the capability to make new aspaces by the idea that you have an existing empty, clonable handle
22:02:00 <geist> the 'create new things by cloning existing things only' is also an interesting route to go down. we ddn't take that route in zircon
22:02:00 <geist> but if you explore it you can also come up with interesting patterns
22:03:00 <Griwes> yeah
22:03:00 <Griwes> alright, I have plenty of food for thought now, thanks!
22:03:00 <geist> but still it's fundamentally a capability style route
22:03:00 <geist> sure thing!
22:03:00 <geist> i thin kthe trick is tocome up with some paradigm and then try to map it to everything so you end up with a consistent pattern
22:03:00 <geist> you might have to make exceptions here or there, but its still a nice design if you can mostly build something consistent
22:04:00 * Griwes nods
22:05:00 <tusko> please to help tusko :(
22:05:00 <tusko> i will be your friend forever
22:06:00 <Griwes> Alright I think this discussion has pushed me into a specific direction, which means I can get unblocked on actually writing OS code
22:14:00 <geist> Griwes: yay!
22:14:00 <geist> tusko: okay, hang on, moving to another room will take a look
22:15:00 <tusko> thank you so much!
22:17:00 <geist> tusko: in general you dont need to copy from user since hte output is overwritten
22:18:00 <geist> otherwise looks okay. not sure if you have anything in particular you want a commentary on
22:18:00 <tusko> I guess it just didn't work when I compiled it
22:19:00 <tusko> I was concerned about issues of isolation between userspace memory and kernel memory
22:19:00 <tusko> That test program reports back 1 + 2 is 0
22:19:00 <geist> well, i dont know exactly what it takes to add a linux syscall, which is what this appears to be
22:19:00 <geist> but the overall strategy of what you're doing here looks okay
22:19:00 <geist> if you want actual 'how do i do this in linux' this isn't really the right channel
22:20:00 <geist> as an overall 'how syscalls work generically on kernels' this seems reasonable
22:20:00 <geist> like in this case is the result pointer actually pointing to user space? duno. i'd try printing it
22:20:00 <geist> also make sure the arg direction is correct in copy_to_user
22:21:00 <tusko> Ok. Yeah, I'm not perfectly sure either. I definitely have a handle on how to add it to the kernel, but the mechanics of passing that pointer around and writing it are a little opaque to me
22:21:00 <geist> yah might be intersting to see what the absolute value of the pointer is
22:21:00 <geist> maybe there's some layer above in the kernel that makes a temporary for you that result is actually pointing at?
22:25:00 <geist> honestly the fact that it has naked pointers to stuff in user space is pretty scary
22:25:00 <geist> seems so easy to accidentally violate the rules of not touching things directly in user space
22:29:00 <tusko> I think I was actually missing a header <linux/uaccess.h> also. I saw some errors I hadn't noticed before while building about implicit definition of those copy functions
22:35:00 <tusko> its pretty shitty having to wait like 30m per compile. how do you cope as an OS developer?
22:41:00 <geist> oh eep. yeah fix the warnings
22:51:00 <tusko> that seems to have done it. I also changed the copy calls to copy_from_user(&tmp, result, sizeof(int)) and copy_to_user(result, &tmp, sizeof(int)) respectively
23:51:00 <heat> hi sortie
23:52:00 <sortie> Jeez the heat is onto me
23:52:00 <heat> do you have any advice on porting packages that use cmake? without the environment leaking
23:52:00 <sortie> This is a the winning move is not to play type situation and I won
23:53:00 <heat> i've wrapped pkg-config but I see there's still some weirdness with the way cmake finds packages and the toolchain file
23:53:00 <heat> dunno if you've seen stuff like this
23:53:00 <sortie> Would you believe I literally don't have any ports using cmake?
23:53:00 <heat> wowzers
23:54:00 <sortie> So I don't know much about cmake but I get the impression a lot of the interface and logic is offloaded to people's cmake scripts so the behavior is probably going to vary wildly to the point that it's not a standard interface?
23:54:00 <sortie> (Where ./configure is a fairly standardized interface although ports sometimes do not quite play by the rules and I fix that)
23:55:00 <heat> hmmm not so much, cmake has a bunch of defined interfaces to find packages and libraries and whatnot
23:55:00 <heat> like budget pkg-config
23:55:00 <sortie> What problematic behavior exactly are you seeing with cmake?
23:55:00 <heat> and cross compilation is pretty well defined
23:56:00 <heat> the host environment is leaking
23:56:00 <heat> I think I found out why but i'm not too sure
23:56:00 <graphitemaster> you just chroot to build every package
23:56:00 <graphitemaster> that's the only sane way
23:56:00 <heat> issue: I'm building the least trivial package, llvm
23:57:00 <sortie> The general debugging pattern is to find a value that's wrong and try and trace it back to where it was determined, and see at what stage there's a confusion between build/host/target
23:58:00 <heat> from a quick google before I went to bed last night I think I found out that cmake is really stupid when find_package() is involved
23:58:00 <heat> like, it uses your sysroot but if it can't find the package it tries the host libraries *for some reason*
23:59:00 <sortie> graphitemaster, I hear you, though that's also coming from a perspective of “oh man gotta herd all these upstream cats”, and the alternative philosophy I take is to nail down all of these cross-compilation bugs (and they are bugs, conceptual bugs, not using the right variables, doing wrong stuff, etc.) and actually fixing them
23:59:00 <sortie> So yeah I gotta get why de-facto Linux distros would want to chroot such builds because stuff is fucked yo
23:59:00 <sortie> *totally get