Search logs:

channel logs for 2004 - 2010 are archived at http://tunes.org/~nef/logs/old/ ·· can't be searched

#osdev2 = #osdev @ Libera from 23may2021 to present

#osdev @ OPN/FreeNode from 3apr2001 to 23may2021

all other channels are on OPN/FreeNode from 2004 to present


http://bespin.org/~qz/search/?view=1&c=osdev&y=19&m=8&d=12

Monday, 12 August 2019

10:34:31 <Nizumzen> what text editor / IDE do you guys use for OS dev stuff?
10:34:45 <Nizumzen> I've just installed Clion with the NASM plugin
10:55:48 <zid`> vi
10:56:16 <asymptotically> i use emacs with evil :D
11:00:34 <klange> i wrote my own editor
11:00:45 <zid`> in which editor?
11:00:50 <klange> vim originally
11:01:44 <Mutabah> vim
11:02:09 <Nuclear_> vim
11:02:14 <klange> But with all the rewriting and redesigning I've been doing lately, I would say my editor is pretty much made in my editor these days.
11:03:31 <klange> I'm in the middle of some pretty extensive changes to add key rebinding and change how the whole editor works on a fundamental level
11:16:02 <aalm> vim
11:16:07 <aalm> .theo
11:16:08 <glenda> You come off as pretty uncharitable.
11:16:24 <bauen1> neovim (it's just bit faster than vim)
11:18:43 <aalm> gvim (it's just easier than vim)
11:20:38 <j`ey> vim (it's just vim)
11:20:53 <klange> this is what I'm working on at the moment https://klange.dev/s/Screenshot%20from%202019-08-12%2020-19-42.png
11:22:01 <zid`> congratulations it's a source file
11:25:27 <klange> Currently all my keybindings are hard-coded switch statements.
11:25:52 <Nuclear_> I don't see anything wrong with that :)
11:30:55 <bauen1> klange: just use a c compiler to dynamically compile and link the switch statement and you have easy, turing complete configurable key-bindings \o/
11:33:14 <xenos1984> vim + youcompleteme with clang completer + airline
11:33:43 <klange> my editor has some ctags completion, nothing too fancy, just basic symbols
13:04:57 <Nizumzen> hmm I think Clion has a vim plugin
16:14:31 <doof_> hey everyone, i am a new member in this channel. It would be great if u guys could help me into getting started with os dev. I do have some theoretical knowledge but never tried making an OS.
16:15:26 <aalm> welcome, which arch?
16:15:36 <doof_> arch?
16:15:40 <aalm> .theo
16:15:40 <glenda> You don't know what you are talking about, and you don't seem to have the ability to wrap your mind around all the parts that are involved.
16:15:57 <aalm> x86?
16:16:01 <doof_> oh yes
16:16:43 <doof_> where should i start?
16:16:54 <Bitweasil> At the beginning. How much x86 do you know?
16:17:01 <Bitweasil> At a "how the hardware works, operating modes, etc" level?
16:17:46 <doof_> from an OS perspective i know about how instructions are executed and can also understand assembly code
16:17:49 <aalm> doof_, familiar w/osdev wiki ?
16:18:02 <doof_> only heard of it, never read
16:18:04 <Bitweasil> Do you understand real mode, protected mode, ISRs, etc?
16:18:17 <doof_> i know about kernel mode and user mode
16:18:34 <Bitweasil> So, no.
16:18:35 <Bitweasil> https://software.intel.com/en-us/articles/intel-sdm#three-volume
16:18:41 <doof_> yeah no
16:19:10 <Bitweasil> You'll need to understand a good bit of Volume 1, but the good stuff for OS dev is Volume 3.
16:19:13 <Bitweasil> About the first half of it.
16:19:26 <Bitweasil> The OSDev wiki is good, but assumes you know your way around the weeds of x86 first.
16:19:37 <doof_> looks good, thanx
16:20:00 <Bitweasil> If you're going to do a real mode OS, you can skip a good bit of SDM3, but if you want something in 64-bit long mode, about half of it's relevant, maybe 2/3rds. You can skip the virtualization chapters.
16:20:11 <Bitweasil> start reading.
16:20:39 * Bitweasil waits patiently for doof to realize just how many pages long those wonderful bits of arcane wisdom are. ;)
16:21:01 <doof_> ok will do, so how long before i can start coding
16:21:06 <aalm> .theo
16:21:07 <glenda> Who didn't solve that problem? People like you.
16:21:33 <Bitweasil> How fast do you read? If you don't know what you're trying to do or how to get there, coding won't really help you much.
16:21:58 <Bitweasil> If all you want to do is copy-paste some code you don't really understand from some tutorials into a file that you can run in an emulator, that's not OS Dev.
16:22:06 <Bitweasil> That's cargo cult coding.
16:22:12 <Bitweasil> (at best)
16:22:30 <doof_> ok got it
16:22:36 <doof_> will start right away
16:22:41 <doof_> thanx
16:22:58 <Bitweasil> You're looking at months to something that's a limited kernel, years to get good at the ring 0 magic.
16:23:34 <Bitweasil> A decade or so should suffice to get you competent in a lot of the realm, though there will still be plenty of dark corners that are "Oh, yeah, I know they're out there..." level knowledge. ACPI is one of those dark corners to me. :/
16:23:41 <doof_> will take a look at Bare_Bones as well
16:24:14 <doof_> yeah its a looot of reading
16:24:40 <Bitweasil> It's a metric shitton or twelve of complexity.
16:25:05 <Bitweasil> The good news is that if you get good at it, very few people play in this realm and you should be able to leverage the knowledge into a rather nice bit of getting paid.
16:25:13 <Bitweasil> The bad news is that I'm not kidding about a decade to get good at it.
16:25:29 <Bitweasil> A few years covers a lot of territory, but a modern computer is horrifyingly complex.
16:26:15 <doof_> well lets see how far i can go
16:26:31 <Bitweasil> Enjoy. It's a fun space.
16:26:39 <Bitweasil> For... well, you have to have madness immunity, really.
16:26:39 <doof_> how do mention someone in this chat
16:26:47 <doof_> @ is not the way to do it
16:26:51 <Bitweasil> Dunno. It's IRC.
16:26:59 <doof_> ok cool
16:27:03 <Bitweasil> Usually just type their name and their client will work it out, it's not part of the IRC spec.
16:27:13 <doof_> ohk
16:27:23 <aalm> usually ppl have highlighting on their nick
16:27:53 <doof_> yeah i noticed
16:27:57 <Bitweasil> Yeah, but that's just a client convenience, not actually a "mention" or stuff. IRC predates all of that.
16:27:58 <Bitweasil> Thankfully.
16:28:07 <aalm> yep
16:28:34 <doof_> so why do people not move on from irc
16:28:40 <Bitweasil> It's "just" text based, low bandwidth chat in the backwaters of the internet.
16:28:42 <Bitweasil> Because it works.
16:28:48 <Bitweasil> And because it's reasonably n00b-unfriendly.
16:28:55 <doof_> hahaha
16:28:59 <doof_> thats true
16:29:06 <Bitweasil> So you get a pretty solid concentration of highly skilled and technical people in an environment where they aren't being bothered by the outrage of the week.
16:29:15 <Bitweasil> And because the bandwidth requirements are basically zero.
16:29:46 <Bitweasil> Which is useful as a lot of the highly skilled people have retired to the hinterlands without gigabit internet connections.
16:29:50 <doof_> how long people here doing os dev, on an average
16:29:51 <doof_> ?
16:29:59 <Bitweasil> You can use IRC over a cell link without trouble and without insane bandwidth fees.
16:30:07 <Bitweasil> Not-so-hypothetically for a number of people I know on IRC.
16:30:19 <Bitweasil> Oh... I've been at it most of a decade.
16:30:30 <Bitweasil> On and off a bit, call it 6-8 years of hands on work in ring 0 or below.
16:30:42 <doof_> whats ring 0?
16:30:45 <Bitweasil> ...
16:30:49 <Bitweasil> Get reading.
16:30:53 <matt|home> <Bitweasil> A few years covers a lot of territory, but a modern computer is horrifyingly complex. <-- there's a very good reason why "the linux kernel" wasn't actually entirely written by torvalds. there's a dedicated team who built the whole damn thing, he just set up the foundation. if you can call it that.
16:31:25 <Bitweasil> Sure, and the computer is horrifying without touching the first set of device drivers.
16:31:27 <matt|home> doof - yeah seriously start reading. in all seriousness if you're interested in os development i really would recommend being proficient in C
16:31:29 <Bitweasil> Those are broken.
16:31:34 <matt|home> in C first*
16:31:47 <Bitweasil> IIRC the IDE controllers were a particularly horrifying bit of madness.
16:31:53 <Bitweasil> (the old PATA stuff)
16:32:08 <Bitweasil> You had ones that were just, by any reasonable definition, "flat out broken." They literally didn't do what they were supposed to.
16:32:12 <Bitweasil> And the drivers had to work around it.
16:32:12 <doof_> thanx matt, will do
16:32:24 <Bitweasil> You had "DMA Capable" ones that would massively corrupt DMA transfers.
16:32:35 <Bitweasil> So you had to know that those ones weren't able to be used with DMA, despite advertising it.
16:32:37 <Bitweasil> Etc.
16:33:36 <Bitweasil> But overall, the Linux Kernel is a pretty nice place to play around. You can write modules that do things without having to worry about a lot of the details.
16:33:54 <Bitweasil> (useful for playing in ring 0 to do minor learning tasks)
16:34:03 <matt|home> i think my biggest complaint by far is that while i appreciate backwards compatibility, there are a LOT of archaic non-standardized weird methods that x86 used and continue to use, whereas simpler architectures like for microcontrollers are much more standard and straightforward to use.
16:34:11 <Bitweasil> Certainly.
16:34:22 <Bitweasil> You need 40 years of back knowledge to make sense of a modern x86 machine.
16:34:25 <matt|home> so that means if you want to be a serious OS developer for like a company or something, there's just sooooo much bullshit you have to wade through
16:34:44 <Bitweasil> Because crap like the PIC and A20 line and such show up just often enough you have to recognize them.
16:34:56 <Bitweasil> (not the APIC, the legacy daisy chained PIC setup)
16:35:14 <matt|home> do you know how UEFI handles the a20 stuff? if it does?
16:35:27 <Bitweasil> Then you've got the IOAPIC, but also message signalled interrupts, and to make sense of that, you have to read ACPI. :(
16:35:37 <Bitweasil> I think it enables the line and quietly tries to forget it exists.
16:35:50 <matt|home> right.. what does it do to enable it. i read there's like six different methods
16:35:51 <Bitweasil> But I'm about 95% certain that it still exists on current gen hardware.
16:36:00 <Bitweasil> If I had to guess? All of them until one works.
16:36:13 <Bitweasil> Check the open source UEFI code, see what it does.
16:36:23 <Bitweasil> Unless an early binary blob from Intel handles that.
16:37:11 <Bitweasil> https://github.com/tianocore/edk2/search?q=A20&unscoped_q=A20 <-- Take a look at some of those results.
16:37:17 <Bitweasil> They look relevant.
16:37:19 <matt|home> im working on a bootloader and kernel for an older intel machine, so legacy bios.. but eventually i wanna take a look at the UEFI stuff later. im glad it got standardized
16:37:49 <Bitweasil> //
16:37:49 <Bitweasil> // Disable A20 Mask
16:37:49 <Bitweasil> //
16:37:49 <Bitweasil> IoOr8 (0x92, BIT1);
16:37:57 <matt|home> yeah i see
16:38:30 <Bitweasil> Looks like the Fast A20 Gate toggle.
16:38:51 <Bitweasil> The legacy way involves the keyboard controller, just like everything else on the platform.
16:39:10 <matt|home> gotta love history .. "we're short a pin! hey, the keyboard has one free let's use that!" what does the keyboard have to do with any of this.. NOTHING.
16:39:16 <Bitweasil> Exactly.
16:39:44 <Bitweasil> So, you know how the 286 didn't have a way to go back from protected mode to real mode?
16:39:46 <Bitweasil> Without a reset?
16:39:56 <matt|home> luls, no but i do now
16:40:07 <Bitweasil> The "official" way to go back was to reset the platform with the keyboard controller reset command.
16:40:14 <matt|home> -
16:40:15 <matt|home> -_-
16:40:23 <Bitweasil> But for OS/2, for DOS multitasking stuff, they were doing that a ton and it was slow.
16:40:28 <Bitweasil> There's another, far faster way to reset the CPU.
16:40:40 <Bitweasil> So OS/2 would just triple fault the CPU to reset it to real mode. Far faster than using IO ports. :)
16:41:17 <matt|home> i get that backwards compatibility is important.. but man x86 is really bad for learning a useful subject in IT :\
16:41:29 <Bitweasil> https://en.wikipedia.org/wiki/Triple_fault#Other_uses
16:42:12 <Bitweasil> http://www.os2museum.com/wp/loadall-strikes-again/ is another fun read on weird-ass stuff OS/2 did.
16:42:52 <Bitweasil> http://www.os2museum.com/wp/himem-sys-unreal-mode-and-loadall/ also
16:43:29 <matt|home> lemme ask you, for something with similar capabilities to DOS, how much memory do you think is sufficient? im talking basic file system access, and theoretically networking. can get away with 512 MB ?
16:44:18 <Bitweasil> Well, 640kb was enough for an awful long time.
16:44:34 <Bitweasil> If you can't do it in a couple meg, you're doing something pretty wrong.
16:45:13 <matt|home> im just wondering because friggin windows ME needed like 2 gigs minimum to function, and im pretty sure dos with graphics could get away with like 512
16:45:26 <Bitweasil> I am pretty damned sure Windows ME did /not/ need 2GB to run.
16:45:34 <Bitweasil> because I ran 2000 and XP on an awful lot less RAM.
16:46:17 <matt|home> i remember running it on the worst damn desktop i've ever had, it was.. awful
16:46:19 <Bitweasil> I had 640MB of RAM in a box back in the day, and that was luxurious.
16:46:46 <Bitweasil> ME needed 32MB of RAM, minimum.
16:47:23 <matt|home> im gonna make a claim and say that's a dirty dirty lie by the marketting team.
16:47:38 <Bitweasil> Sure, but ME never required 2GB of RAM.
16:47:45 <Bitweasil> 2GB of RAM was literally impossible on most of the boards that ran it.
16:47:48 <mrvn> matt|home: AmigaOS worked with 256kB of memory and was a full graphical multitasking system.
16:48:08 <Bitweasil> (you didn't have enough RAM sockets for the RAM of the day to put 2GB in, even if you wanted to)
16:48:10 <mrvn> Something like DOS should work with 64KB ram
16:48:28 <Bitweasil> Eh, he did say networking.
16:48:33 <Bitweasil> That's pretty hard to do on 64kb.
16:48:38 <mrvn> Bitweasil: did you say DOS?
16:48:42 <matt|home> gotcha..
16:48:48 <Bitweasil> Since you have to reassemble fragments and such.
16:48:59 <Bitweasil> <matt|home> lemme ask you, for something with similar capabilities to DOS, how much memory do you think is sufficient? im talking basic file system access, and theoretically networking. can get away with 512 MB ?
16:49:06 <mrvn> MTU = 1500. you never need more ram than that.
16:49:07 <Bitweasil> And I said a few megabytes would be sufficient for that.
16:49:23 <Bitweasil> Yeah, unless you want to actually reassemble a longer datagram.
16:49:23 <matt|home> mrvn - i said im working on something with similar capabilities to dos, maybe with networking and graphics. i haven't used super super super old machines, so i don't know what they needed for most of that
16:49:32 <Bitweasil> Remarkably little.
16:49:46 <Bitweasil> You could do Windows 3.1 with a couple meg of RAM, with full networking.
16:49:48 <mrvn> AVRs with 16kb ram can do ethernet
16:50:04 <Bitweasil> Yes, but it tends to be somewhat limited in what they can actually handle. it's not a full on networking stack.
16:50:20 <Bitweasil> I've done it, certainly, but you can only do a tiny bit of network communication - it's not a proper network stack.
16:50:20 <mrvn> Bitweasil: they can't handle multiple frames at once. So it's slow
16:50:37 <Bitweasil> And they usually don't even handle multiple streams.
16:50:43 <mrvn> forget raising the window size to 8MB
16:51:04 <Bitweasil> They can do ethernet/TCP/IP communication, but it's not really anything I'd call "networking support" beyond the most basic "I can talk to a single remote endpoint" level.
16:51:13 <Bitweasil> But with a few meg, you could have a real network stack.
16:51:25 <mrvn> Default buffer for a tcp socket is like 64KB in normal OSes. So good luck doing that with 16kb ram. :)
16:51:38 <Bitweasil> Yeah.
16:51:57 <mrvn> Still. if your code is well written you can make it scale to what memory you have.
16:52:05 <Bitweasil> Again, in the context of "reasonable networking support," you could do it on 640kb, but a few meg is a lot nicer.
16:52:13 <matt|home> hm. suggestions for what i could implement with a full 4 gigs of ram? i haven't really thought about scope and scale in this sense
16:52:19 <Bitweasil> Or just run Trumpet Winsock. :)
16:52:31 <matt|home> i only wanted to make something basic and functional, but if i have the resources.. might as well use them.
16:52:46 <mrvn> matt|home: run frefox for short times
16:52:51 <Bitweasil> From a small OS perspective, the difference between 4GB and infinite is academic.
16:53:09 <Bitweasil> Though in 32-bit mode, you're not going to address 4GB of RAM - the PCI memory mapped regions take up half a gig, typically.
16:53:16 <mrvn> Bitweasil: The difference bewteen <= 4GB and >4GB is huge
16:53:19 <Bitweasil> And PAE is way worse than just going to long mode.
16:53:31 <Bitweasil> mrvn, in the context of a simple OS development task? No, they're not.
16:53:37 <Bitweasil> For running a modern OS, sure.
16:53:37 <matt|home> listen, i have a personal hatred for modern web browsers now -_-
16:53:39 <mrvn> Bitweasil: sure. different architecture
16:53:55 <matt|home> wow half a gig for the mapped stuff? i thought it was a few kb at most :\
16:54:02 <mrvn> Bitweasil: with <= 4GB being much harder.
16:54:15 <Bitweasil> Depends on the hardware, but usually you drop about 500MB of RAM up at the top of the 4GB space.
16:54:25 <Bitweasil> No big deal if you are in long mode, but for 32-bit, it's tricky to work around.
16:54:32 <mrvn> matt|home: pci cards can have a lot of ram and register space
16:54:35 <Bitweasil> mrvn, why is 32-bit harder?
16:54:46 <Bitweasil> Plus you've got the APIC and such mapped up there.
16:54:51 <matt|home> bah
16:55:06 <Bitweasil> There's just a lot of memory mapped stuff that ends up in the last half gig or so before the 4GB mark.
16:55:22 <mrvn> Bitweasil: because with 4GB ram in 4GB address space you can't simply designate large sparse regions for jobs. Can't just say everything >= 0x80000000 is kernel space.
16:55:51 <Bitweasil> I thought that's exactly what Windows did back in the day...
16:55:54 <Bitweasil> But, yeah, fair point.
16:56:03 <Bitweasil> You can make some nice shortcuts in ia32e mode.
16:56:12 <Bitweasil> Of course, that leaves you vulnerable to Meltdown and ilk. ;)
16:56:23 <mrvn> Bitweasil: can't just map all physical memory at 0xFFFF000000000000
16:57:09 <mrvn> Bitweasil: Anything above 2GB ram I would insist on 64bit mode.
16:57:18 <Bitweasil> In /any/ case, for a toy OS project, it doesn't matter because you're using megabytes, not gigabytes.
16:57:29 <Bitweasil> And if you are using gigabytes, it's no longer a toy OS.
16:59:27 <mrvn> sure. But if you have it would you write the OS to not use it? WOuld you simply ignore anything above 1GB?
16:59:50 <Bitweasil> It depends. What am I writing an OS for?
17:00:11 <mrvn> It's the path that counts, not the destination. :)
17:00:15 <Bitweasil> if I'm writing it to learn low level stuff, absolutely, I'd ignore it and build a small toy that doesn't bother with anything up there.
17:00:29 <Bitweasil> If I wanted to do anything productive, I'd not write my own OS in the first place, and use an OS that has proper support. :)
17:00:49 <Bitweasil> The "path" has a mountain of bodies at the first few steps in.
17:01:05 <Bitweasil> So few people make it to even a basic real mode "OS" (DOS capability) that there's no point in arguing about what's further down the path.
17:02:34 <Bitweasil> brb, need more coffee.
17:14:55 <mrvn> In vacuum particles particles randomly appear out of the quantum foam. If the universe is infinite doesn't that mean that at any moment an exact duplicate of you will random appear somewhere, at least for a little while till the anti-you hits it.
17:14:59 <mrvn> ?
17:32:45 <eryjus> mrvn, unless you are next to a blank hole, where one of you might be sucked into the black hole and the other you might escape and go on...
17:33:03 <eryjus> s/blank/black/
17:35:09 <mrvn> eryjus: but then you would be going at the speed of light so time would stand still.
17:35:33 <mrvn> .oO(Who wants to live forever? *sing*)
17:36:24 <xenos1984> "live" is rather relative if you stand still in time
17:36:44 <eryjus> ahhh.. not it's not the speed of light that make time stand still on the event horizon... it's the infinite gravity well.. but agreed about living forever
17:41:27 <mrvn> eryjus: traveling at the speed of light your mass is infinite. your are your own gravity well
18:28:09 <eryjus> mrvn, agreed, but I also thought that gravity had an impact on space/time; where infinite gravity resulted in infinite time.... perhaps I am recalling Science Channel shows wrong...
18:54:39 <j`ey> anyone know how to jump to a high half / new VA in arm64?
18:55:12 <j`ey> tried something simple like this: https://paste.rs/Cvi.s
18:57:32 <xenos1984> j`ey: what about this instead of the first line? ldr x0, =foo
18:57:50 <xenos1984> that's what I use
18:59:52 <j`ey> do I need to have removed the 1:1 pa<->va mapping before that?
19:00:41 <xenos1984> no, actually you still need it until the jump
19:00:52 <xenos1984> you can remove it afterwards
19:01:28 <xenos1984> (of course you need to have the higher half mapping, too - already before the jump)
19:02:26 <j`ey> actually, im confused what Im trying to do :D
19:03:25 <eryjus> j`ey: blx
19:03:30 <eryjus> let me get an example
19:04:44 <j`ey> I have my kernel at phsy 0x40000.., I made a new mapping from va 0x8000.. to physx 0x40000
19:04:57 <eryjus> this is the jump from my loader: https://github.com/eryjus/century-os/blob/master/modules/kernel/src/loader/LoaderMain.cc#L63
19:05:35 <xenos1984> eryjus: blx? Isn't that for branch and link with mode change?
19:05:44 <eryjus> and the actual implementation: https://github.com/eryjus/century-os/blob/master/arch/arm/entry.s#L467
19:06:05 <eryjus> and sorry -- i just loaded pc directly
19:06:35 <eryjus> xenos1984, you are probably right.
19:09:15 <xenos1984> Also that's aarch32 code, right? Probably that works on aarch64 as well, I'm not that familiar with arm assembly.
19:09:20 <j`ey> in this case, I just want to jump to a label or the 'next instruction' in a different va
19:10:26 <j`ey> I suppose I could do something like (addr_of_inst - phys_load_addr) + virtual_load_addr
19:20:13 <xenos1984> Have you tried the method I suggested?
19:22:18 <j`ey> yes, it didn't really change much
19:22:27 <j`ey> I dont really understand how it's actually meant to work!
19:22:30 <xenos1984> BTW, it seems that "br x0" is actually encoded as "sub pc, x0, zr". Which is probably also how "mov pc, x0" would be encoded.
19:23:27 <xenos1984> Well, if foo: is the higher half label, then ldr x0, =foo should load its address in x0. And br x0 jumps there.
19:24:01 <xenos1984> Assuming that your kernel is linked at the higher half VA.
19:24:20 <j`ey> im just not sure how to make that label the "higher half" if that makes sense
19:25:28 <xenos1984> Ah, I see. That should be via the linker script. The whole section with that label needs to have a higher half VA set in the linker script.
19:25:59 <j`ey> oh
19:26:39 <eryjus> j`ey, are you 100% assembly?
19:26:48 <j`ey> no
19:27:35 <j`ey> xenos1984: can you link your code again?
19:28:10 <eryjus> ok, you are going to have to create a loader text section (say, ldrtext) and a kernel text section (say, krntext) and the same with data sections. Then you will need to decorate your code so that they are placed in the correct sections and your linker script locates them properly.
19:29:18 <eryjus> this is likely going to force you to revisit your PIC problem. Also, make sure you do not read loader data from kernel and kernel data from the loader. same with calling kernel functions from the loader.
19:29:21 <xenos1984> https://github.com/xenos1984/NOS/blob/master/kernel/vendor/raspi/Entry.a64.S
19:31:22 <j`ey> xenos1984: thanks
19:31:40 <j`ey> eryjus: Im building without PIC currently
19:34:10 <eryjus> i believe it will work (don't quote me) until you start setting up data in the kernel space from the loader or try to call a kernel function (not your jump, mind) from your loader. In the loader you will have to ensure that your kernel is mapped properly to where your linker script set it all up.
19:36:17 <j`ey> im going to try the (hacky?) approach I said above: (addr_of_inst - phys_load_addr) + virtual_load_addr
19:49:48 <j`ey> https://paste.rs/mGe.s this seems to work
19:53:09 <j`ey> I dont know how 'unusual' it is though
19:53:17 <xenos1984> indeed, if you want the return branch to work when this is inside a function, you need to modify lr as well
19:53:31 <j`ey> I did ^
19:53:43 <xenos1984> yes, I noticed
19:55:25 <j`ey> I dont know what downsides this might have
19:56:40 <xenos1984> mostly you need to be careful about using the correct address of symbols (VA vs PA)
19:57:32 <xenos1984> before the jump you address everything with the VA = PA mapping
19:58:11 <j`ey> Im going to try modify my page tables (globals inside Rust), I dont know how well that willw rok
19:58:14 <xenos1984> after the jump you want to use the higher half mapping, so VA = PA - KERN_PHYS + KERN_VA
19:58:14 <j`ey> *will work
19:59:25 <xenos1984> no idea how well Rust can manage with changing the VA in the middle of the program and how to link that properly...
20:11:03 <j`ey> looks like it might have some issues, but I will have to solve them another time
21:00:50 <griddle> Im kind of having trouble wrapping my head around the whole "map the page tables into the 511th index of themselves" idea
21:02:51 <griddle> I mean, I understand the mapping of the page directory, but that only works for the top level, right?
21:14:45 <Bitweasil> Context?
21:16:20 <heat> griddle: I imagine you're using x86_64?
21:16:29 <heat> if so, I don't think that's usually done
21:16:43 <Bitweasil> It only makes sense for ia32e, just... not sure I've ever seen it.
21:16:54 <Bitweasil> Of course, I may not have explicitly looked for that.
21:17:20 <heat> usually you just map all physical memory and then to access page tables/random phys memory you just do PHYS_BASE + paddr
21:17:52 <Bitweasil> At least on Linux. Windows does it slightly differently with the various pools.
21:18:28 <heat> yeah I don't know how Windows does that so I can't help you with that
21:19:14 <heat> Do note that usually PML4[511] is occupied with the kernel mappings
21:19:41 <griddle> So what is a common x86_64 way of reading and writing page tables?
21:19:57 <heat> griddle, told you already
21:20:53 <griddle> If I want to avoid mapping all of physical memory?
21:21:06 <heat> dunno
21:21:22 <Bitweasil> If you're in ia32e, why would you want to avoid mapping all physical memory?
21:21:25 <Bitweasil> You've got the space.
21:21:34 <Bitweasil> Otherwise you'd need something to touch physical memory as needed, or... get creative.
21:21:36 <heat> there's no disadvantage to having a big mapping
21:21:44 <griddle> im assuming you mean to map memory with huge pages, right?
21:21:49 <heat> yes
21:22:00 <heat> I use 2MB as a fallback and 1GB when available
21:22:43 <Bitweasil> You don't have to, but there's no reason /not/ to. And many valid reasons to do so.
21:23:49 <heat> also my page allocator automatically zeroes pages unless you explicitly tell it not to
21:24:02 <heat> (and it uses the phys mapping too)
21:24:37 <griddle> my physical page allocator currently has a 'scratch page' that is mapped 1:1 that when it wants to read/write to a page, it maps it there
21:24:53 <heat> yeah but that's quite terrible for the TLB
21:25:02 <griddle> yeah, doesnt scale either
21:25:14 <heat> and it susceptible to races between CPUs/whatever
21:25:23 <heat> *it's
21:25:39 <griddle> so I guess I need to rebuild my paging system to support mixed page sizes?
21:26:20 <heat> griddle: https://github.com/heatd/Onyx/blob/master/kernel/arch/x86_64/paging.c
21:26:26 <heat> that's how I do my paging stuff
21:26:34 <heat> note that's there's a bunch of terrible code
21:27:20 <griddle> eh, cant be worse than mine :)
21:27:41 <Bitweasil> Yeah... that whole scratch page thing. If you're going to do that, use PCIDs.
21:27:48 <Bitweasil> Otherwise you're blowing your TLB constantly.
21:27:54 <heat> mostly as I developed my OS throughout the years I kinda forgot paging code because it's mostly useless
21:28:01 <Bitweasil> With PCID and INVPCID, should be fine.
21:28:08 <Bitweasil> Just... you know, /use/ that.
21:28:35 <heat> you just need the whole "map page, unmap page, change prots, get phys address of mapping" interface
21:28:58 <griddle> Yeah, Im also having problems with, when mapping a page, causing a page fault trying to read a page dir
21:29:52 <heat> that's what you were trying to solve with a recursive mapping (and what I solve with the direct physical mapping)
21:30:15 <griddle> I guess before I do the full physical mapping, I need to still do the scratch mapping, right?
21:30:34 <heat> yes
21:30:49 <heat> mostly
21:31:31 <griddle> or, because my boot assembly id maps the first 512 4k pages, I could instead just make a new address space, map the memory in with 1G pages, then switch to that new address space
21:31:51 <heat> don't do that
21:32:07 <heat> Map the kernel initially with 2MB pages
21:33:05 <heat> I do switch address spaces at a point but it's when I enforce protection of kernel segments
21:33:18 <griddle> the reason I map with 4k pages is because my paging code didn't support any other page sizes, so when modifying the boot page dir I would get a bunch of panics
21:33:29 <heat> you don't do those mappings in assembly?
21:33:33 <griddle> I do
21:33:55 <griddle> but my C code doesnt understand 2mb pages, so it breaks
21:34:21 <griddle> 1BG pages in in CPUID, right?>
21:34:36 <heat> you just had to look at the huge page bit and the paging level where it's at
21:34:38 <heat> yes
21:38:54 <griddle> I also had some questions about how processes are laid out in memory. I know that processes all have the kernel code mapped into them, right?
21:39:03 <Bitweasil> Not anymore, no.
21:39:06 <Bitweasil> Meltdown.
21:39:13 <Bitweasil> But, traditionally, yes.
21:40:58 <heat> Realistically, you want to do that
21:41:08 <griddle> okay, so then if I ignore meltdown, the kernel being mapped into process address space means processes need to exist entirely above physical memory, virtually
21:41:09 <heat> because we really kinda don't care about meltdown
21:41:19 <heat> what?
21:41:43 <heat> Wait, you're talking about linux kernel stuff
21:41:50 <griddle> nah, not linux
21:42:05 <heat> why are you talking about above physical memory?
21:42:13 <heat> that's what linux does in i386
21:42:37 <griddle> like, If I 1:1 map physical memory in the kernel, then I dont want a process clobbering that, right?
21:42:53 <heat> You divide the address space in two halves
21:43:16 <griddle> so like `[ kernel space ] [ process space ]`
21:43:46 <heat> then you have a space in the middle that isn't available because it's non-canonical(x86_64 cpus don't implement the full 64-bit of addressing because it's just wasteful, so you end up with a bunch of space in the middle that GPFs when you try to address it)
21:44:06 <heat> usually it's [user space] - [kernel space]
21:45:04 <heat> do note that x86_64 manages to do that canonical/non-canonical stuff by using the top bits as an indication of whether it's in the top or bottom of the address space
21:45:25 <griddle> so then the kernel code is mapped to a different location than its physical location? Are kernels commonly position independent?
21:45:51 <Bitweasil> Generally, yes.
21:45:54 <heat> 1) yes 2) depends, but real kernels do that
21:46:04 <Bitweasil> The Linux kernel executes with kernel virtual addresses.
21:46:17 <Bitweasil> sudo cat /proc/kallsyms
21:46:23 <Bitweasil> Do that on a Linux box and you'll see the address space it uses.
21:46:31 <Bitweasil> Which on 64-bit systems is in the upper canonical half.
21:46:39 <heat> then, basically, the top half of the address space is mapped in every process and the kernel does with it what it wants to do with it
21:46:54 <Bitweasil> Yes, as supervisor pages.
21:46:56 <heat> so you end up with a kernel space mmap that's more versatile because you're in the kernel
21:47:04 <heat> and that's basically it
21:47:18 <griddle> oh yeah, wow. Lots of high addresses, huh
21:47:26 <heat> yes
21:47:51 <heat> griddle: that's how linux does it https://www.kernel.org/doc/Documentation/x86/x86_64/mm.txt
21:49:00 <heat> Oh, also, note that kernel pages are regularly mapped as global as to not thrash the TLB
21:49:20 <heat> (and supervisor so the user can't access them)
21:52:54 <griddle> okay, so I think I kind of understand this, please correct me if Im wrong... the kernel maps itself into every process at a really high virtual address, so the process can have the rest
21:53:20 <griddle> the process can virtually have addresses 0 to -23 TB or whatever
21:53:24 <heat> really high = exactly at the start of the second half
21:54:15 <heat> note that the kernel image itself is usually mapped at around -2GB
21:54:18 <griddle> okay, so then if the kernel wants to write to physical memory, does it convert the physical address to `(second_half) + pa`
21:54:25 <heat> no
21:54:29 <griddle> oh
21:55:08 <heat> it converts the physical address to PHYS_BASE(where phys_base is the base address of your direct memory mappings, that you chose previously) + pa
21:55:41 <griddle> oh okay
21:55:54 <heat> quick tip: don't abuse that direct memory mapping thing too much since it's really insecure and bug-prone
21:56:16 <zid`> I use it in one location, zeroing pages in the allocator
21:56:22 <heat> better to page fault then to overwrite other data and have random bugs that you can't really figure out
21:56:56 <zid`> and it's terrabytes away from kernel and user data, so hard to accidentally generate a pointer to, hopefully
21:57:05 <heat> Oh, and don't assume that memory that's virtually contiguous is also physically contiguous, because it normally really isn't
21:57:42 <griddle> yeah, but It is probably fine to have physical memory mapped 1:1 so I can maintain my physical freelist or whatever, right?
21:57:55 <heat> yes
21:58:13 <griddle> as long as I'm not using physical memory directly as a heap or something, I should be fine
21:58:13 <heat> it's in those kinds of situations that you wan't to use the physical map thingy
21:58:18 <heat> want*
21:59:26 <griddle> I guess that second half concept makes alot of sense when you think about elf image loading
22:00:08 <heat> it makes even more sense when you consider that normally the user space is oblivious to those kinds of details
22:00:22 <griddle> (randomized address space aside), you can just read it into memory where the elf expects to be
22:00:44 <Bitweasil> Did you read the Linux mm.txt file? It explains the layout quite well.
22:00:52 <heat> Note: mmap doesn't read files into memory
22:00:55 <Bitweasil> And, yes, you put the elf at the expected virtual address.
22:00:58 <heat> at least in modern linux
22:00:59 <Bitweasil> And it simply works.
22:01:03 <heat> /unix
22:01:09 <griddle> mmap is lazy, right?
22:01:13 <heat> mmap is doubly lazy
22:01:16 <griddle> reading the file when you trigger a page fault within a region
22:01:37 <Bitweasil> mmap is lazier than Garfield.
22:02:04 <griddle> how so?
22:02:05 <heat> file mappings usually fault in a COW read-only page that's shared with the page cache, that then is copied and faulted in when when you try to write to it
22:02:26 <Bitweasil> ^^ Not only does it not read the file in, it may not even bother putting the mappings in.
22:02:36 <Bitweasil> It just makes a note of them so when you /do/ fault in that region, it can do the right thing.
22:02:39 <heat> anon mappings just map a COW zero page and do that same thing
22:02:50 <heat> Yes, that's why I said faulted in
22:03:06 <Bitweasil> I think there's a way to hint "I would like this actually paged in" with madvise? But even large mallocs don't map everything in.
22:03:16 <heat> yes
22:03:28 <Bitweasil> I don't recall the point at which it goes to a delayed allocation, but even in the kernel, a large vmalloc isn't guaranteed to actually exist.
22:03:32 <heat> MADV_WILLNEED
22:03:36 <griddle> mmap is magic
22:03:56 <Bitweasil> Man, userland code is so nice. :)
22:04:01 <Bitweasil> I remember not having to worry about all this stuff.
22:04:08 <griddle> ha, I remember writing javascript
22:04:10 <heat> "mmap is just a collection of magic tricks" - heat
22:04:20 <griddle> "whats a heap" is a phrase I remember saying
22:04:35 <Bitweasil> heat, and everything, under the hood, is just mmap. ;)
22:04:42 <heat> exactly
22:04:46 <heat> that's the beauty of it
22:04:49 <Bitweasil> ... for a depressingly wide range of "everything" :/
22:05:02 <heat> in the linux kernel inodes have their own address space
22:05:24 <heat> so shared mappings are really just mappings in your process' address space that point to another address space's mapping
22:05:53 <heat> Is it a crap abstraction? yes
22:05:57 <heat> Do they care? no
22:06:24 <griddle> Im assuming linux also shared the page directories for shared mappings as much as it can?
22:06:42 <heat> no
22:06:56 <heat> Same thing with tasks = processes except when they're not because they retrofitted threading into the kernel and user-space
22:07:40 <heat> griddle: generally you don't share that stuff because of things like virtual address alignment and dirty bits, etc
22:08:13 <griddle> okay, so back to the mm.txt thing
22:08:36 <griddle> Im assuming that linux, internally, has a macro like `v2p` and `p2v` that converts physical to kernel virtual, right?
22:08:41 <heat> mm.txt isn't really helpful
22:08:44 <griddle> by just adding that -23TB number
22:09:21 <heat> it's actually 0xffff888000000000
22:09:39 <Bitweasil> Yes...-ish. Except that you can't go from any virtual address to a physical address, only... is it kmalloc?
22:09:52 <griddle> doesnt the virtual address hardware truncate addresses to 48 bits or something?
22:09:53 <Bitweasil> I think kmalloc gives you stuff you can convert to physical, vmalloc doesn't.
22:10:07 <Bitweasil> Depends on the hardware. But somewhere in that range, on mainstream chips, yes.
22:10:16 <heat> But if you want my honest opinion, there are loads of linux kernel concepts/code that are really really garbage
22:10:23 <Bitweasil> Some are 39, some are 48, new stuff with 5 level paging is coming out with 57 bit VA.
22:10:35 <heat> Bitweasil: yes, kmalloc is physically contiguous, vmalloc isn't guaranteed to be
22:11:02 <Bitweasil> Linux is an ugly collection of hacks - but it's also one that works.
22:11:17 <Bitweasil> There are plenty of ways you can do this stuff, limited mostly by your creativity and how it crashes into the hardware.
22:11:25 <Bitweasil> Certain concepts just don't work well on x86.
22:11:37 <Bitweasil> (the "physical access page" thing, before PCID, is an example of something that's just awful in performance)
22:11:50 <Bitweasil> And PCID without INVPCID was... odd.
22:12:09 <heat> what I consider the saner approach is to vmalloc everything
22:12:23 <Bitweasil> That works up until you need physical regions for hardware interfacing.
22:12:29 <Bitweasil> DMA or such.
22:12:34 <heat> uh, yes it does
22:12:36 <griddle> Im not too familiar with the linux internals, what do {k,v}malloc do?
22:12:41 <griddle> phys/virt?
22:12:42 <Bitweasil> ... which is it's own train wreck, and the IOMMU can help some.
22:12:52 <zid`> <heat> Bitweasil: yes, kmalloc is physically contiguous, vmalloc isn't guaranteed to be
22:12:52 <Bitweasil> It depends on the DMA engine and what it can put up with.
22:12:55 <heat> Bitweasil: I just allocate enough pages and that's it
22:13:46 <Bitweasil> Also, kmalloc won't typically get you large regions long after boot.
22:13:49 <Bitweasil> vmalloc can.
22:13:57 <heat> yes, that's the thing
22:14:07 <zid`> which is a problem if you support hotpluggable drivers
22:14:09 <heat> I don't like my allocations failing
22:14:28 <heat> zid`, modern hardware is generally decent enough to not require that sort of things
22:15:05 <heat> I'm very conviced that if my vm subsystem runs out of memory the kernel crashes and burns
22:15:23 <zid`> heat: My ethernet card at least, wants 16kB physical buffers for DMA etc
22:15:38 <zid`> I just use a single 2MB page and put 2MB/16kB buffers on it
22:16:21 <heat> That's just 4 pages and you don't have those crappy requirements like alignment or being below 16MiB, etc
22:16:52 <zid`> it's still not 4k though
22:16:59 <zid`> so it requires contiguous pages
22:17:03 <zid`> or a 2MB page
22:17:08 <heat> yes
22:17:21 <zid`> so hotplugging drivers for example, *needs* you to be able to provide that at any point in time
22:17:39 <heat> but of course, you also have IOMMUs and whatever that can get you out of that sticky situation
22:18:14 <Bitweasil> If you have them.
22:18:21 <Bitweasil> They're more common, but still far from everywhere.
22:18:29 <heat> Yes, that's why I only support modern hardware
22:18:35 <heat> Screw legacy stuff
22:30:06 <griddle> position independent code works for data as well, right?
22:30:30 <AndyHD> Is there any way to develop an OS without c/c++?
22:30:58 <griddle> in rust :)
22:31:01 <_mjg> ;0
22:31:04 <_mjg> yes and no
22:31:19 <_mjg> if you are asking, you should probably try a c kernel first
22:31:41 <_mjg> mostly because virtually all examples of how to do something will be in this lang
22:31:51 <heat> griddle: yes
22:31:56 <AndyHD> I really dislike c...
22:32:10 <heat> it works by addressing things based on the current rip
22:32:22 <_mjg> AndyHD: i don't blame you
22:32:27 <heat> AndyHD: What language are you thinking of?
22:32:28 <griddle> if you want to write a kernel in something other than C, you will probably needa a runtime system in C or asm
22:32:51 <AndyHD> heat: I would like to have an signularty OS with just one programming language
22:33:02 <_mjg> AndyHD: you wont escape at least reading at lot of c anyway and quite frankly first kernel in something other than c sounds like a suicide mission
22:33:02 <heat> but which language?
22:33:33 <AndyHD> Kotlin/Java... and I know that the runtime is OS devendent... that's the issue so far
22:33:55 <griddle> hmm, porting the JVM to a kernel sounds like alot of C in the background
22:34:04 <zid`> so you want to write a JVM in C first then
22:34:08 <zid`> and kernel to host it
22:34:16 <zid`> good luck have fun
22:34:18 <griddle> then bindings you can call from java
22:34:45 <heat> well, if you really want to do that then you just want to AOT or JIT code
22:35:01 <heat> because interpreting code is slooooooooooooooooooooooooooooooooooowwwwwwwww
22:35:17 <AndyHD> heat: AOT
22:35:52 <heat> But honestly? Neither of those languages is suited for kernel development
22:36:10 <AndyHD> the Java code does not need to be platform independent in this case
22:36:31 <heat> My honest recommendation is to write the kernel in C/C++ and write the rest in Java/kotlin
22:36:38 <heat> it's much much more doable
22:36:41 <heat> look at android
22:37:09 <isaacwoods> you could write the platform bit in C/C++/Rust/something else and use JNI to talk to it from java maybe
22:37:28 <heat> if you do it that way the only non-jvm code you need is the kernel and the jvm itself
22:37:34 <heat> which is actually quite okay
22:38:03 <griddle> and alot of runtime. You'd basically need to implement posix to get something like openjdk on t here
22:38:10 <heat> yes
22:38:21 <griddle> which, honestly, is most of kernel development
22:38:31 <heat> but it's likely you could write an okay one yourself
22:38:50 <griddle> I have put thought towards writing an OS where user processes are just lisp programs
22:38:51 <heat> meh, not really
22:39:01 <AndyHD> Too bad ... I thought it would go otherwise ... if c still exists... I can also use the linux kernel...
22:39:02 <zid`> oh no he's retarded :p
22:39:03 <griddle> can't run binary, code
22:39:32 <griddle> you can totally do this with a linux kernel module
22:39:34 <heat> AndyHD: Yes, using the linux kernel is a very good alternative
22:39:49 <heat> (in your case)
22:39:58 <AndyHD> heat: and that's the way I don't like it
22:40:13 <AndyHD> I would like to have a micro services...
22:41:01 <griddle> funny you should mention that, cause I am working on a research paper in which a specialized hypervisor and OS is used for microservices
22:41:30 <heat> you just have to face that java crap isn't used or even helpful at lower levels
22:41:41 <griddle> haven't gotten too far, still in the proof of concept stage
22:41:43 <heat> Disclaimer: I really like C/C++
22:42:17 <griddle> C is great because it isnt limited by a runtime
22:42:17 <heat> you can write it in rust if you wanna be a hipster
22:42:27 <AndyHD> on my current linux box there are so many libaries installed... from so many different programming languages....
22:42:29 <AndyHD> In the meantime, I just do not like this mixture of thousands of libraries anymore.
22:42:34 <heat> or go
22:42:48 <griddle> honestly, more languages need to be embeddable
22:43:06 <griddle> like, it would be amazing if I could throw go code into my kernel and all I provide is mmap
22:43:10 <griddle> for example
22:43:10 <heat> you cant escape the libraries
22:43:19 <heat> what
22:43:34 <griddle> go is already statically compiled, right?
22:43:42 <heat> yes?
22:43:45 <heat> what about it?
22:44:09 <griddle> so it would be nice if I could remove the posix part of the go runtime, but keep the GC stuff in there by just providing a mmap interface
22:44:22 <griddle> then you could embed it
22:44:36 <heat> you do know that you can load whatever into your kernel if you can provide it in a relocatable way
22:45:17 <griddle> yeah, but what im saying is more languages should provide embedding features like lua or haskell have out of the box
22:45:22 <heat> griddle: then you would need to provide replacements for the go runtime
22:46:02 <heat> the thing is that lua (dunno about haskell) is a completely different language from java, kotlin, c++, c#
22:46:16 <heat> lua is a scripting language
22:46:19 <griddle> yeah
22:46:35 <griddle> but what Im saying is, if you want to embed haskell, you call an `Hs_Init()`function
22:46:44 <griddle> and i think you can provide function bindings somehow
22:46:48 <heat> so have multiple runtimes builtin?
22:47:04 <griddle> have a system that runtimes can be swapped by the embedder
22:47:33 <heat> note that I'm currently trying to mix C and C++ code in my C kernel and it's not easy at all
22:48:02 <griddle> yeah
22:48:19 <griddle> I do hate that c++ has different semantics about void* though
22:48:55 <heat> I don't
22:48:59 <heat> It's on purpose
22:49:07 <heat> void * is not type safe
22:49:29 <griddle> yeah, I agree, but it makes the interfacing between c and c++ messy :)
22:49:31 <zid`> C++'s void * semantics are silly
22:49:42 <heat> zid`, why?
22:49:50 <zid`> The only thing it can do is hide bugs
22:50:08 <zid`> it can't ever prevent one
22:50:15 <heat> hide bugs when?
22:50:32 <zid`> when the expression you cast accidentally didn't have void * type like it was supposed to
22:50:51 <zid`> in C that'd be a type issue because you were relying on the implicit conversion from void * and there wasn't one
22:51:03 <zid`> in C++ you've got the cast there so it'll just go "7 to int *? sure"
22:51:41 <heat> conversions from integers whose size != pointer size are at least warnings
22:52:19 <heat> AND if you're misusing the interface that much, maybe it's not that good of an interface
22:52:43 <heat> (how do you confuse numbers with a pointer??)
22:52:51 <zid`> ..or any other pointer type
22:53:08 <heat> and casting is generally avoided
22:53:17 <zid`> int to void * can happen in C, so it hides a real bug with not including stdlib and trying to use malloc
22:53:30 <zid`> because of the implicit declaration nonsense
22:53:52 <zid`> a *lot* of people used to C++ will do char *a = (char *)malloc(10);
22:54:07 <zid`> which is a genuine bug if you forgot to include stdlib.h, but will crush the warning
22:54:12 <heat> yeah but those people don't know how to program in C
22:54:13 <griddle> `auto a = (char*)malloc(10);` :)
22:54:27 <zid`> griddle: is meaningless in C and there's a no such type as char*
22:55:06 <zid`> heat: C's semantics are less error-prone and the code is nicer to look at and easier to maintain, specifically for void *. Even if it is all super marginal.
22:55:17 * heat does T *p and T& r
22:55:28 <heat> Is it less error-prone?
22:55:46 <zid`> I gave an example of a bug you can only cause with C++'s semantics?
22:56:38 <heat> type safety exists so you can't easily mistake types
22:56:52 <zid`> exactly, and C is better here :P
22:57:14 <zid`> void * exists precisely to bypass type-safety, but then C++ makes it so you can cause more bugs
22:57:22 <zid`> because you have to supress warnings as you do it
22:57:28 <heat> bypassing type-safety is not good
22:57:35 <heat> if you want to bypass it, use a cast
22:57:35 <zid`> and yet C++ didn't remove it
22:57:44 <zid`> the cast *supresses warnings*
22:57:47 <zid`> how is that safer
22:58:04 <heat> casts make it so you think about the type conversion that you're doing
22:58:12 <heat> implicit = bad in this case
22:58:45 <heat> note: C style casts are kinda frowned upon in C++
22:59:08 <zid`> you just argued against yourself there ^
22:59:23 <heat> Did I?
22:59:33 <zid`> C++ forces you to do C style casts
22:59:34 <zid`> C does not
22:59:37 <heat> No
22:59:45 <zid`> (which is funny)
22:59:53 <heat> C++ introduces static_cast, dynamic_cast, const_cast which are safe, and reinterpret_cast, which is unsafe
23:00:06 <zid`> so why not make void * work only with them
23:00:14 <heat> Uh, const_cast is also unsafe
23:00:19 <zid`> except it didn't, so it didn't introduce them did it
23:00:22 <zid`> it just made a C semantic worse
23:00:25 <griddle> const_cast is the reason why const use useless
23:00:41 <griddle> is useless'
23:00:47 <heat> void * still works with casts -_-
23:00:53 <zid`> no, it *only* works with casts
23:00:56 <zid`> it shouldn't need casting
23:01:02 <zid`> because that's a way to hide a bug
23:01:09 <heat> <zid`> so why not make void * work only with them
23:01:18 <heat> it does only work with them
23:01:27 <zid`> except it doesn't
23:01:35 <zid`> I can just cast it
23:01:42 <zid`> no need for your underscores
23:02:02 <heat> C-style casts are usually a big red flag in a lot of code bases
23:02:11 <zid`> so why allow them
23:02:12 <heat> except mine cuz I don't care :P
23:02:22 <heat> compatibility
23:02:29 <zid`> C++ is a shitshow, news at 11
23:02:50 <heat> and C isn't?
23:03:31 <heat> let me implicitly declare shitshow() and hide actual bugs that actually happen
23:05:10 <zid`> yep, implicit declarations are also nonsense
23:05:13 <zid`> but at least I admit it :P
23:05:30 <zid`> so is NULL being allowed to be declared as 0, and a host of other things
23:06:33 <heat> C is nonsense
23:06:48 <heat> the fact that there hasn't been a new standard since C11 is nonsense
23:06:54 <zid`> There's a decent core language in there, some of the fiddly details is stupid
23:06:59 <zid`> heat: since C99*
23:07:08 <zid`> all C11 did was deprecate VLAs :P
23:07:08 <heat> the fact that they don't try to improve it at all is nonsense
23:07:19 <zid`> yea well all the standards committe members are also the C++ members
23:07:20 <heat> and add atomics, threads
23:07:25 <zid`> all they do is collect handmedowns
23:07:40 <zid`> Plan9 has had some very very good shit for decades
23:07:42 <zid`> but you can't use that
23:08:00 <heat> like what?
23:08:36 <zid`> I believe it has the feature I really really want and designed myself and then found out plan9 had: Dereference pointers to structs containing incomplete members
23:08:43 <griddle> If I am going to make an enum for mapping sizes, 2mb = ::big_page, 1gb = ::huge_page, what should 4k be?
23:08:45 <griddle> small page?
23:08:53 <heat> griddle, page
23:08:56 <zid`> I just have 4K_PAGE, 2M_PAGE :P
23:08:57 <griddle> o
23:09:13 <heat> s/big/large/
23:09:25 <griddle> Can you have macros starting with numbers?
23:09:39 <heat> no
23:09:45 <zid`> yes
23:09:48 <heat> yes?
23:09:51 <zid`> because macros are just text-replacement
23:09:51 <griddle> huh, til
23:09:54 <zid`> they have no semantic meaning
23:10:10 <j`ey> #define 6FOO, like that?
23:10:16 <xenos1984> #define 2 3
23:10:43 <heat> https://godbolt.org/z/lBW2tm
23:10:45 <heat> doesn't work
23:11:45 <zid`> it wants valid identifier names, rip
23:11:46 <zid`> how rude
23:11:58 <zid`> get a less anal preprocessor :p
23:12:24 <griddle> okay, code style question
23:12:26 <griddle> `void map(u64 va, u64 pa, u16 flags = 0, pgsize size = pgsize::page);`
23:12:33 <griddle> should I flip flags and size around?
23:12:48 <zid`> flags should be last imo
23:13:05 <zid`> and size should follow the thing it's the size of, the u64s
23:13:05 <griddle> and default to rwx?
23:13:30 <heat> don't use u64 and u16
23:13:39 <heat> use uint16_t or uint64_t
23:13:48 <heat> no
23:13:48 * zid` just has #define u16 etc in his headers
23:13:55 <griddle> internally, its defined to that
23:14:01 <griddle> uint64_t is just an ugly type
23:14:11 <zid`> yea perfectly cromulent
23:14:18 <heat> that's why I use unsigned long
23:14:26 <heat> like a man of culture that I am
23:14:30 <zid`> unsigned long is too much typing and might be the wrong size in this instance
23:14:37 <griddle> imagine using two words for a type, blech
23:14:41 <heat> unsigned long = uintptr_t
23:14:52 <griddle> `unsigned long long int *` is the best type
23:14:59 <heat> and uintptr_t is ******ugly******
23:15:09 <griddle> macos and linux define uint64_t to be different things
23:15:14 <griddle> so printf complains
23:15:28 <griddle> "use %lld" on macos, "use %ld" on linux
23:15:40 <zid`> that's non-standard, griddle
23:15:48 <zid`> so stop doing non-standard things and you won't get bitten
23:15:52 <zid`> PRIu64 is the proper method
23:16:07 <heat> if unsigned long != word size then get a proper platform
23:16:14 <zid`> Which is ugly as fuck, of course
23:16:32 <griddle> `unsigned long long == unsigned long`
23:16:35 <griddle> on mac
23:16:50 <heat> so uint64_t is unsigned long?
23:16:57 <griddle> doesnt the spec say that long long just needs to be at least as wide as long?
23:17:38 <griddle> no, I think uint64 is u long long
23:17:54 <griddle> and uint32 is unsigned long, for some reason
23:18:05 <heat> if uint64_t = unsigned long long and unsigned long long = unsigned long then it's fucking obvious that uint64_t = unsigned long
23:18:06 <zid`> griddle: This is what the ILP32 etc names mean
23:18:16 <zid`> whether integers longs and pointers are 32bit or 64bit
23:18:34 <griddle> just use void* for all data and let that be the end of it
23:18:47 <griddle> need an int? void*
23:18:58 <zid`> like C++ :P
23:19:20 <griddle> so llvm doesnt have a type for a generic pointer
23:19:27 <heat> nah
23:19:28 <heat> use __m512i
23:19:43 <griddle> I do like that llvm lets you just have a type `i2048`
23:19:55 <heat> CPU doesn't support AVX512? G A R B A G E
23:20:16 <zid`> mine doesn't :(
23:20:24 <zid`> heat buy me a 9900k
23:20:29 <griddle> https://godbolt.org/z/8kBzjm
23:20:57 <zid`> til there are people who can just type out llvm ir
23:21:00 <griddle> 6000 lines of assembly
23:21:12 <griddle> ive spent alot of time working on languages lol
23:21:21 <heat> griddle: returning 2048 bytes is exactly the kind of code that javascript programmers would write
23:21:33 <heat> zid`, no
23:21:35 <griddle> or beginner c devs
23:21:47 <heat> zid`, why don't you buy ME a 9900k
23:21:51 <griddle> `char[] foo()`
23:21:56 <zid`> because you wouldn't appreciate it
23:22:11 <griddle> "why does returning a stack array not work?"
23:22:34 <heat> zid`, yes I would
23:22:39 <heat> i promise
23:22:47 <chrisf> how much does the 9900K cripple itself when you actually use AVX512?
23:22:49 <zid`> heat: Not as much as I would
23:22:54 <zid`> chrisf: idk I never found out
23:23:00 <zid`> I did ask my friend who buys a lot of xeons for work
23:23:01 <griddle> okay so buy two, one for you and one for heat
23:23:09 <heat> yeah exactly
23:23:37 <zid`> that means I am down a 9900k rather than up one
23:23:40 <zid`> math doesn't check out
23:23:47 <griddle> how many registers does x86 have now?
23:23:54 <zid`> several
23:24:16 <griddle> also, I love how much work goes into deciding if a process in linux needs to save the xmm registers
23:24:22 <heat> griddle, 15 GPR
23:24:37 <heat> + %rsp + %rip + all the xmm ones + whatever
23:24:44 <griddle> so like, a million
23:24:45 <zid`> griddle: they don't anymore afaik they juse xsave
23:24:52 <heat> griddle, it's like 0 work
23:24:52 <zid`> because the logic is too much overhead compared to just doing it
23:24:57 <heat> ^^
23:24:59 <griddle> xsave writes a full page, right?
23:25:05 <heat> uhm, no?
23:25:10 <zid`> it's just 'pusha'
23:25:13 <zid`> for the modern regs
23:25:26 <griddle> idk, kvm's api takes a page in as an argument
23:25:32 <griddle> thats my only experience with it
23:25:38 <zid`> well kvm has special requirements
23:25:56 <heat> just, checked
23:25:57 <heat> https://github.com/heatd/Onyx/blob/master/kernel/include/onyx/fpu.h
23:26:05 <heat> it's half a page
23:26:11 <griddle> so when I get around to writing my scheduler, I should just xsave
23:26:12 <heat> and needs to be 64-byte aligned
23:26:33 <heat> Probably the most legacy I support in my kernel is supporting FPU without xsave
23:26:53 <zid`> griddle: Are you counting MSRs?
23:27:11 <griddle> I still love how emulating fpu instructions works on old hardware
23:27:17 <griddle> zid`: idk, what do you mean
23:27:24 <griddle> in a context switch?
23:27:34 <zid`> was it not you who asked
23:27:38 <zid`> someone asked how many regs x86 had now
23:28:03 <griddle> oh, I was being sarcastic, because they keep adding more on
23:28:29 <zid`> There's probably like a meg of state now or something :D
23:28:54 <griddle> I half expected there to be a context switch instruction
23:29:02 <heat> arm64 has 30 GPR
23:29:10 <zid`> 31 is a zero or something?
23:29:17 <zid`> well, 0
23:29:18 <griddle> like, with how much time is spent context switching, I could see intel adding hardware to write the state to cache directly
23:29:32 <zid`> doesn't help
23:29:37 <zid`> the cpu runs faster than that
23:29:48 <heat> switching states is fast man
23:29:55 <zid`> so hardcoding an instruction that probably doesn't do what you want isn't useful
23:30:02 <zid`> it can decode all your pushes faster than it can actually push them
23:30:26 <zid`> especially because if you're doing a lot of syscalls, the syscall entry is probably in icache
23:30:32 <heat> the only "slow" part of context switching is TLB switching and stuff
23:30:40 <heat> and that's actually not that slow
23:30:43 <zid`> and yea, if your code causes a tlb miss, you're waiting for that anyway
23:30:45 <griddle> just have more TLBs
23:30:47 <griddle> :)
23:30:48 <zid`> so who gives a fuck
23:31:40 <heat> I'm just waiting for quantum computing just so I can break cryptography
23:33:16 <heat> those tiny little invisible bosons and fermions WILL break your encryption and let me access your 30GB hentai folder
23:33:20 <heat> and there's nothing you can do
23:34:23 <griddle> 30GB? weak
23:34:26 <griddle> think PB
23:34:49 <zid`> think goats
23:35:01 <heat> lebron and mj?
23:35:09 <heat> sorry I had to make this joke
23:35:17 <heat> don't hit me pls
23:35:19 <zid`> those are basket hoop players right
23:35:27 <griddle> does it commonly happen that you need to map a page into another address space?
23:35:42 <heat> MJ is retired, lebron is 34 years old and still kicking it
23:35:51 <heat> griddle: depends on your API
23:35:59 <griddle> wait yeah, when you make a new process
23:36:06 <griddle> Im thinking having two main functions
23:36:08 <griddle> `void map_into(u64 *p4, u64 va, u64 pa, pgsize size, u16 flags);`
23:36:21 <griddle> and the a `void map(...)` that doesnt take the p4
23:36:21 <heat> in POSIX you generally don't need to
23:36:25 <zid`> all mappings are into another address space except kernel to kernel :P
23:37:25 <heat> but if you make an API like zircon where processes are created empty and the parent is responsible for mapping executables, creating threads, and starting them, then it's likely you'll need that
23:38:03 <griddle> how does the global mapping thing work?
23:38:22 <heat> I'm doing a mixed approach on my second kernel where I provide a process creation function that can do stuff for you but also not do stuff for you
23:38:33 <heat> griddle: you set the G bit, and you're done
23:38:47 <heat> the TLB isn't flushed on context switch for those entries
23:38:49 <zid`> just erm, don't pretend it isn't true
23:38:54 <zid`> else you might have issues
23:39:00 <zid`> they WILL be global :p
23:39:05 <griddle> so I could have kernel virtual memory mapped globally like that
23:39:36 <heat> like what
23:39:43 <griddle> globally
23:39:59 <heat> yes that's how classic x86 kernels do it
23:40:17 <griddle> all PML4s contain the same upper half, if I do the linux kernel thing
23:40:18 <zid`> pre security researchers ruining our lives
23:40:26 <griddle> screw those guys
23:40:34 <griddle> im never going to get enough users for it to be a problem
23:40:53 <heat> nowadays with meltdown and KPTI they have two top-level paging dirs, one with the kernel mapped and another without it mapped
23:41:25 <heat> and then your ints, irqs and syscalls go to a guard page that switches dirs
23:41:46 <griddle> but I could have a function that creates a PML4 by cloning one with a zeroed out lower half
23:42:22 <heat> that's what I do
23:42:26 <griddle> cool
23:42:43 <griddle> also, im thinking of going full object oriented with my paging system
23:42:51 <griddle> like, defining c++ classes for each of the layers
23:42:53 <heat> I don't recommend that
23:42:57 <griddle> no?
23:43:05 <zid`> overheads will kill it if it doesn't any actually OOPy :P
23:43:13 <zid`> and it doesn't help you write it
23:43:14 <heat> how are you going to allocate memory while you're allocating memory?
23:43:36 <griddle> `this` is a pointer to the page location
23:43:39 <heat> you want to minimize those kinds of allocations
23:44:02 <griddle> I wouldnt actually allocate, or have any virtual functions
23:44:14 <griddle> and none of the classes would have fields
23:44:31 <griddle> it would just be applying types to the page directory layers, and statically knowing what you can and cant do with them
23:44:37 <heat> hmmm I'm not sure if the code generated would be decent
23:44:48 <heat> since you would need placement new
23:45:17 <griddle> eh, reinterpret cast :)
23:45:28 <zid`> My bootloader paging code is great it does double casts
23:45:30 <heat> oh, right
23:45:33 <zid`> beat that
23:45:40 <heat> zid`, double casts?
23:45:45 <zid`> yep!
23:45:52 <heat> what do you mean?
23:45:54 <zid`> It truncates the address then casts it to a pointer
23:46:58 <griddle> ha, g++ complains too much if I do that, so I have done `(T*)(u64)(u32)ptr`
23:47:00 <griddle> to truncate
23:47:09 <zid`> triple cast? you beast :D
23:47:10 <zid`> https://github.com/zid/boros/blob/master/boot/main.c#L100
23:47:14 <griddle> no wait, g++ would complain with that
23:47:14 <zid`> I've only got a double you win
23:47:22 <griddle> because you are losing precision ptr->int
23:47:32 <griddle> `(T*)(u64)(u32)(u64)ptr`
23:47:36 <griddle> youd need to do that, I think
23:47:45 <griddle> at that point, just reinterpret_cast
23:48:03 <heat> why are you truncating pointers
23:48:23 <zid`> because the page tables are 64bit but the code is 32bit
23:48:31 <zid`> this is the protected -> long bootstrap
23:48:42 <griddle> in c/
23:48:44 <griddle> ?
23:48:48 <zid`> ofc in C
23:48:52 <griddle> I do that in asm
23:48:52 <zid`> who'd use java
23:48:56 <griddle> lol
23:48:58 <griddle> so topical
23:49:23 <zid`> https://github.com/zid/boros/blob/master/boot/long.asm#L28 That's my assembly for setting up paging
23:49:27 <griddle> I do the 2mb identity paging in asm. Just a couple lines
23:50:19 <heat> it's doable in asm
23:50:29 <zid`> I'm not writing an ELF loader in assembly
23:50:52 <zid`> I should hoist 2 of those lines into C and do 2MB pages, but meh
23:51:03 <zid`> that'd require finding the editor with that file open
23:51:19 <griddle> do you reuse the elf loader for process loading later?
23:51:23 <zid`> nope
23:51:29 <griddle> I guess cause its 32 bit
23:51:40 <heat> In my second kernel I load myself from my uefi bootloader and then the kernel just relocates itself
23:51:43 <griddle> c++ should let you template on bit count
23:51:45 <zid`> that and it'd be really gross to link a file from boot/ into the kernel :P
23:52:04 <heat> griddle: and you can
23:52:07 <griddle> ooh
23:52:12 <heat> it's called
23:52:16 <heat> using uint*_t types
23:52:33 <griddle> but then gcc emits `rsp` and `rbp`
23:52:39 <heat> what
23:52:42 <zid`> what
23:52:57 <griddle> sorry, I mean templating a function to be 32bit or 64bit
23:53:01 <zid`> what
23:53:23 <heat> yeah that seems like a disaster
23:53:28 <zid`> how is that even possible
23:53:43 <zid`> you compile it -m32, the template generates different things, but then it generates 64bit code?
23:53:44 <zid`> WHAT
23:53:57 <heat> you'd have to mix 32-bit and 64-bit code in a .o
23:54:21 <griddle> templates typically live in header files
23:54:36 <griddle> its the same source code, but gets compiled for your 32bit and 64 bit
23:54:42 <griddle> wait nvm, that would already happen with -m32
23:54:47 <griddle> ignore me
23:57:20 <griddle> what do you guys call the page directory layers internally. I feel like calling them PML4, PDP, PD and PT is just too much
23:57:30 <heat> call it PML
23:57:39 <griddle> so like PML4, etc..
23:57:44 <heat> no, just PML
23:57:49 <heat> they're all identical
23:57:59 <zid`> I call them struct page_table
23:58:16 <zid`> but you knew that
23:59:48 <Bitweasil> PML4, PDPT, PD, PT.