channel logs for 2004 - 2010 are archived at http://tunes.org/~nef/logs/old/ ·· can't be searched
#osdev2 = #osdev @ Libera from 23may2021 to present
#osdev @ OPN/FreeNode from 3apr2001 to 23may2021
all other channels are on OPN/FreeNode from 2004 to present
http://bespin.org/~qz/search/?view=1&c=osdev&y=19&m=5&d=11
01:17:31 <stisl> hi
01:18:02 <stisl> eryjus, it looks more and more that our problem could be sth. todo with old pages in memory
01:19:55 <stisl> a few days ago doug_16k told sth. about it that intel is sometimes reflushing the cache on an interrupt which is really slow
01:20:39 <stisl> and when I have two threads: a consumer thread and a producer thread this could be the issue
01:21:33 <stisl> so, the solution is maybe invalidating the pages so that the interrupt is not doing it for us
01:22:33 <stisl> when the thread is not sharing memory with another thread everything works fine
01:22:52 <stisl> also when paging is deactivated everything is working fine
01:23:13 <stisl> and on the emulator also everything is working fine because paging is not / or not correctly implemented
01:25:08 <stisl> doug_16k, could this be an issue?
01:28:50 <stisl> so when I understand this right, the interrupt page flush or however it is called is really slow in practice, so that it only should be only used when you can't manually invalidate the pages
01:30:20 <stisl> I think there was also some option in qemu to correctly simulate the TLB, but I can't remember which one :(
01:30:46 <stisl> they just deactivated it at default because of performance issues
01:32:40 <stisl> this f**ing tlb is making me crazy sometimes, it is awesome sometimes but sometimes really stupid
01:35:38 <stisl> I hope that it will not be a blocker for my RTOS in the future. If I must deactivate it in order to get realtime - I swear you an embedded system can then do a lot the things faster and this without MMU
01:40:39 <stisl> Swapping to the harddrive is forbidden in realtime systems, so when the TLB is making similar or nearly the same issues like swapping I will give up
01:45:28 <zid> except they take nanoseconds not milliseconds
01:46:39 <stisl> some things on the PC are definitely faster in my measurements
01:46:57 <stisl> but the jitter is making me trouble
01:48:27 <stisl> I've never seen a jitter like this before
01:50:16 <stisl> but I hope it is because I am not calling invplg in the producer task
01:52:27 <chrisf> how much jitter are you talking about?
01:52:33 <stisl> seconds
01:52:41 <zid> nothing to do with the tlb then
01:52:59 <zid> you've got some thundering herd or scheduling or some other issue that is starving something
01:53:34 <zid> The only conceviable way I can see the TLB being implicated is stale tlb entries and after a second enough churn of the tlb happens to drop the stale entries out
01:55:07 <immibis> if a TLB miss took seconds you'd notice your computer stalling all the time
01:55:30 <zid> It'd do literally nothing, I serve thousands per second just fine :P
01:56:25 <stisl> when the threads are not using the same memory everything is working fine
01:56:44 <stisl> but when they share it with a variable or some array I will get the jitter
01:56:46 <zid> sounds more like it could be a cache issue
01:56:49 <zid> than a tlb issue
01:56:53 <zid> i.e you're just polling your L1 cache
01:57:08 <immibis> thread 1 isn't reading the value written by thread 2 for a second?
01:57:12 <zid> until something random happens to evict it from L1 (interrupt or whatever) and you have to fetch a fresh copy
01:57:25 <immibis> or thread 1 isn't running at all for a second?
01:57:55 <stisl> thread 1 is running fine when everything is disconnected
01:58:46 <stisl> but if it is connected, yes thread 1 is blocked
01:59:06 <stisl> for a few seconds sometimes
01:59:15 <zid> add a fence and retest?
01:59:36 <stisl> I tried this 100 times, and got everytime the same rsult
02:02:02 <stisl> but now I am not sure if it is really blocked or just showing the old values
02:09:00 <stisl> hmm, I have deactivated the mutexes in order to make it lock free with ringbuffers
02:09:32 <stisl> I also need sth. better than binary mutexes
02:10:56 <stisl> could this be the issue zid?
02:12:41 <zid> add a fence and retest?
02:12:51 <stisl> you mean mfence?
02:13:08 <zid> If that is the suitable fence, yes, otherwise no
02:13:15 <zid> it's your threading code not mine
02:13:19 <stisl> ok, thanks :)
03:30:27 <doug_16k> stisl I was saying that when page fault happens, it flushes the TLB entry corresponding to the faulting linear address automatically
03:31:12 <doug_16k> stores always occur in order, no need to fence unless there is write-combining
03:31:31 <doug_16k> well, as far as other processors see it
03:32:16 <doug_16k> zid, something random doesn't need to happen. there is a cache coherency protocol that makes caches transparent
03:36:34 <doug_16k> you can read things that are cached on other cpus. your cpu will do a cache coherence transaction to probe other cores. other core will see probe and respond. the line will get sent core-to-core and it will continue with the data correctly transferred into your cache
03:37:24 <doug_16k> if you were writing, then it would read for ownership, getting it into your cache as an exclusive line
03:37:57 <stisl> zid, thanks it helped
03:38:41 <doug_16k> stisl, see log please
03:39:00 <stisl> doug_16k, thanks
03:39:07 <doug_16k> https://freenode.logbot.info/osdev/20190511
03:40:03 <doug_16k> stisl, on x86, multi-cpu memory ordering is nearly magic. all stores are visible in order
03:40:53 <doug_16k> the only races that can still happen are races that can't be avoided no matter what you do
03:41:44 <doug_16k> for example, if CPU A does a store and CPU B does a load, there is no guarantee that CPU B sees that store in a timely manner. what it does guarantee is, if a CPU does stores in a certain order, you won't see later stores completed before earlier stores
03:41:46 <stisl> I think I have a priority inversion too, currently I fixed it with cli during the lock, better I will reschedule it, because the cli makes a big jitter too
03:42:14 <stisl> not so big as before,but in microseconds
03:42:49 <doug_16k> you never fence on x86 unless you are storing to a WC memory type framebuffer
03:42:57 <stisl> ah, ok
03:43:28 <stisl> I use currently the lock instruction for the mutex
03:43:42 <stisl> Is this ok for the framebuffer?
03:43:52 <doug_16k> locked instructions are serializing. that is way beyond a fence. it machine clears
03:44:37 <bcos> stisl: Probably not a good idea to store your mutex in the framebuffer.. ;-)
03:45:05 <doug_16k> typically it won't start a serializing instruction until every preceding instruction completes. and it won't start the following instruction until the serializing instruction completes
03:45:51 <doug_16k> point is: locked instruction is like a super-fence
03:46:28 <doug_16k> it is stronger than a fence
03:46:36 <stisl> ok, I understand
03:47:28 <stisl> This also works just for 2 tasks
03:47:44 <stisl> If there are more producers or consumers it will hang
03:48:43 <doug_16k> is there a link to the code that I missed?
03:49:12 <bcos> For real-time you're probably going to want fancy mutexes that take into account the priority then arrival order of waiters
03:49:53 <doug_16k> what if they change priority though?
03:50:10 <zid> have a list of lists and resort all the lists? :P
03:50:36 <stisl> randomly, lol
03:50:47 <doug_16k> randomly is a great idea
03:50:50 <doug_16k> seriously
03:51:29 <stisl> doug_16k, it is currently not published, the code is still very bad ;)
03:51:43 <doug_16k> awww
03:51:48 <doug_16k> probably not that bad
03:52:16 <stisl> ok, maybe I will do this next time
03:52:22 <bcos> doug_16k: Hard to change priority while waiting
03:52:40 <doug_16k> something else can't change its priority?
03:52:46 <doug_16k> another thread*
03:53:01 <bcos> Maybe - no idea if that's allowed
03:53:07 <doug_16k> prioritized insertion is a good idea though
03:53:24 <doug_16k> just realized that it could either say too bad for my scenario or expensively fixup those
03:53:39 <bcos> (would still be tempted to do a "FIFO linked list of waiters, per task priority" thing; depending on scheduling algo)
03:54:36 <stisl> ok, maybe sth like a cycle free depency tree?
03:55:17 <stisl> but I don't have an idea how to make it cycle free
03:55:26 <bcos> Cycles?
03:55:38 <bcos> stisl: What is the scheduling algo/s?
03:55:48 <stisl> currently round robin
03:55:59 <stisl> but it is too dumb implemented I think
03:56:01 <bcos> Ah, yes - hard real-time with round robin...
03:57:27 <stisl> bcos, on a simple datalogger system I made before it wors perfectly
03:57:48 <zid> hard RT by accident, not by virtue
03:58:07 <bcos> With 10000 normal tasks pounding away at the CPU plus one real-time task?
03:58:40 <stisl> no, not so much tasks ;)
03:59:07 <stisl> one task was a timer task and the rest of it just background processes
04:00:00 <bcos> That sounds more like "2 priority levels, round robin within a priority level"
04:00:53 <bcos> (which would work fine if there's very few high priority tasks)
04:01:01 <stisl> could be the people liked it, I don't know why
04:01:40 <stisl> just some hundred lines of code
04:03:55 <stisl> when I lock sth. I should make the priority shortly high in order too avoid the priority inversion right?
04:04:41 <stisl> -o
04:05:44 <doug_16k> stisl, that's one approach
04:06:19 <bcos> When a mutex is released you'd check if there's any waiters, then (if there is) you'd carefully select which waiter gets the lock next (and unblock them)
04:06:28 <doug_16k> other is to temporarily grant a higher priority to the thread holding a lock
04:06:29 <bcos> ...where this careful selection needs to match the scheduling algo/s
04:07:29 <doug_16k> but wait until a higher priority thing comes along and conflicts with the lower priority thing over the lock
04:08:13 <stisl> maybe I need some mix
04:08:35 <doug_16k> stisl, priority inversion isn't a thing you avoid. it's a feature you add to handle the case where a high priority thing needs something that is being held by a lower priority thing
04:09:21 <doug_16k> ideally that case would never happen, but often things are far from ideal
04:10:33 <doug_16k> if you make it so it gives you higher priority when you are holding a lock, then people will just acquire a lock unnecessarily to jack up their speed
04:12:51 <stisl> What I have learned about real time is that you have to handle wrong data, maybe I have learned it wrong, but this helped also a lot to tune the system
04:13:19 <bcos> Wrong data?
04:13:34 <stisl> for example measurement failures
04:14:09 <bcos> Real time = guarantees about how long something will take. E.g. a real-time system might guarantee that it'll take less than 1234 nanoseconds for a very high priority task to get CPU time (regardless of how many lower priority tasks there are)
04:15:15 <doug_16k> more like 1234ms :P
04:15:29 <doug_16k> realtime doesn't imply fast
04:15:31 <stisl> For this I
04:15:43 <stisl> but mostly low latency
04:16:21 <bcos> Of course that's just one example. Maybe the tasks have deadlines and you guarantee they'll get enough CPU time to meet their deadline. Maybe the tasks need a fixed amount of CPU time (e.g. 100 ms of time per second) and you guarantee they'll get that
04:18:06 <stisl> 100ms should not be a problem when the jitter is 1ms and when I have 10 tasks
04:19:16 <stisl> for audio 1 ms would be excellent, because humans don't feel 1 ms
04:20:36 <bcos> The numbers don't really matter (e.g. could be 1 second of CPU time per day, or 999 nanoseconds per millisecond, or...). The important part is the guarantee
04:20:53 <stisl> sure bcos
04:21:45 <bcos> (and that includes saying "Nope, you can't start an 11th task because that might break the guarantees I gave to the first 10 tasks")
04:21:46 <stisl> for this you need a lot of whitebox and blackbox tests
04:23:15 <stisl> sure, you have to calculate the best- and the worstcase
04:24:38 <stisl> and maybe not too much code in order to keep the calculation simple
04:26:18 <stisl> and you have to instrument your code, in order to proove the whole thing in practice
04:27:07 <stisl> should I take gprof?
04:28:11 <stisl> could be a little bit faster and simpler than a complicated and slow logging system
07:42:46 <geist> meep!
07:49:28 <Kazinsal> meep meep
07:52:30 <doug16k> does anyone get msi interrupts on virtio anymore?
07:52:45 <doug16k> msi or msix
07:55:50 <doug16k> I keep thjnking it must be my code but I don't see how, it enumerates all the virtio caps just fine and never encounters anything except type 9 (vendor type)
07:56:07 <doug16k> pci capabilities I mean
07:56:24 <zid> anymore?
07:56:31 <doug16k> this used to work
07:56:51 <doug16k> I found msi and it worked and everything was very happy
07:57:00 <doug16k> now, clunky horrible PCI pin IRQ
07:57:10 <zid> my qemu provides msi for virtio
07:57:18 <doug16k> what version
07:57:18 <zid> not actually /tried/ it of course, but the config space is there
07:57:32 <doug16k> how do you know msi is there
07:57:38 <zid> I read the cap space
07:57:57 <doug16k> what version of qemu?
07:58:11 <zid> 3.1.0
07:58:18 <doug16k> if I bisect I'll know it was good there
07:58:29 <doug16k> thanks
08:08:07 <doug16k> qemu is giving me a misaligned MCFG record :(
08:08:21 <doug16k> I guess I should assume every ACPI thing is totally misaligned eh?
08:08:51 <zid> I only remember that the root table in the bios needs to be 16b aligned
08:09:01 <zid> is there supposed to be an alignment for the rest
08:09:16 <zid> my current tables are all.. 4b aligned, by the looks of it
08:09:27 <zid> except the root which is 16
08:09:33 <doug16k> there are 64 bit fields
08:09:47 <doug16k> 4b aligned is misaligned
08:09:56 <doug16k> sometimes*
08:10:06 <zid> x86 doesn't care at least
08:10:11 <bcos> It's 80x86 - all alignment is just an optional performance hint
08:11:05 <doug16k> x86 ones are misaligned therefore other architectures have them aligned? not sure I follow
08:11:18 <zid> they're provided by the machine
08:11:21 <zid> the machine is either x86 or it isn't
08:11:57 <doug16k> really?
08:12:06 <zid> I believe so
08:12:45 <zid> I'd be upset if a machine with alignment requirements provided it unaligned, not upset if an unaligned one exists on x86
08:12:47 <doug16k> arm desktops have acpi I believe
08:12:56 <doug16k> unaligned has cost on x86
08:13:15 <zid> like, you can contrive a situation where it matters, but this isn't one
08:13:33 <doug16k> you seem to think I care about the performance
08:14:18 <doug16k> question is whether another arch tables are specified to be possibly misaligned too
08:14:48 <bcos> Today I spent about 2 hours writing a table driven "UEFI device path" decoder. The tables I designed contain a 1-byte "type" and an 4-byte "offset of string". I aligned everything on 2 byte boundaries (deliberately mis-aligning the "offset of string" part) because I felt that reducing the size of the tables was more important
08:15:35 <doug16k> why 2 byte boundaries if misalignment doesn't matter?
08:16:17 <jjuran> What about a 3-byte offset field?
08:16:24 <Kazinsal> it's important to make all of your accesses equally slow
08:16:36 <bcos> doug16k: Everything's a compromise
08:17:24 <doug16k> I'm avoiding misaligned accesses in general
08:18:04 <zid> time to use a u64_from_u8 macro then
08:19:41 <bcos> (ideally, I guess I could've used a 3-byte offset and packed the entries so it's a total of 4 bytes, but developer time is also one of the compromises)
08:20:06 <zid> bcos: align it on a 256 byte boundary
08:20:06 <doug16k> if you memcpy the optimizer will do a misaligned mov for me, but my code is written such that it works with no assumption about how misalignment behaves
08:20:10 <zid> shift the address up
08:20:17 <doug16k> if I*
08:25:41 <Kazinsal> Hmm. Uptime test successful. Left an old build of my system up for 30 days and came back, and it still routes just fine.
08:25:41 <Kazinsal> Oddly slow but looking at the tcpdump of the scp connection that's going through it I think it might be a really small TCP window for some reason
08:26:09 <Kazinsal> Which implies I'm doing something wrong enough in the packet forwarding that TCP is going "uh, this connection *sucks*"
08:26:58 <zid> do you support the windowing stuff
08:28:03 <Kazinsal> I am a router. I just rewrite headers and forward packets.
08:28:14 <zid> ah right
08:30:31 <Kazinsal> Yeah, this is really weird, a really small window is being used
08:32:17 <jjuran> Did you ever expand the window in the first place?
08:32:37 <Kazinsal> I am neither of the endpoints
08:36:17 <Kazinsal> Hmm. Okay, I *am* seeing some window scaling, and the connection speed improves over time, but it's still atrocious
08:36:30 <Kazinsal> Though I wonder if a lot of that is the sheer number of vmexits etc. I'm doing
08:38:37 <Kazinsal> Ugh, yeah, there's some vmware fuckery.
08:38:56 <Kazinsal> Sometimes after rebooting one of the components of the network packets just... disappear.
08:39:14 <zid> vmware has a nice bug with sshd
08:39:47 <zid> I wonder if it's relevent to you also
08:40:02 <zid> You have to set IPQoS lowdelay in your ssh config or you can't connect to anything
08:41:01 <Kazinsal> My setup is: [OpenBSD host 1] -- vSwitch0 -- <Araxes router> -- vSwitch1 -- [OpenBSD host 2]
08:41:43 <Kazinsal> What I really need to do is get a pallet of Optiplexes or something and a few extra e1000 cards...
10:07:23 <bauen1> Kazinsal: is the source code for your network stack available ? (i would be interested in how you organised everything)
10:08:13 <zid> and how many copies does it do? :P
02:58:37 <stisl> zid, doug16k thanks for the tipp with the memory fences, now everything is superfast
02:58:53 <stisl> much better than locks
02:59:03 <stisl> I like it
02:59:52 <stisl> now I am testing it on the real hardware ;)
04:20:24 <alberinfo_osdev_> hi, i have a question. i recently got my OS to 64-bit long mode, but now, i can't search for the rsdp, throws a #GP fault. also, i can't access the mboot header, it gives a page fault, with cr2 set in C000066A. i can't fix the rsdp because bochs gives me access_read_linear(): canonical failure, which says to me that the address that fails is non canonical(0x000F68C1), but without reading the area from 0x000E0000 to 0x0010000
04:21:57 <isaacwoods> alberinfo_osdev_: not a fix to the canonical address issue, but if you're booting via GRUB, why not just ask it to give you the RSDP address
04:22:36 <alberinfo_osdev_> how? it's the first time i hear about that.
04:22:59 <isaacwoods> there's a tag you put in your multiboot header for it and it gives you the correct physical address
04:23:07 <isaacwoods> can't remember the details now, have a look at the spec
04:24:13 <alberinfo_osdev_> ok ill have a look, but what about i cant access the mboot_header? as i said, when i try to access it, throws a page fault with cr2 set around the 3GB mark
04:24:47 <isaacwoods> have you set paging up - how are you getting into long mode?
04:25:22 <alberinfo_osdev_> yes, i map from 0 to 4mb
04:25:56 <isaacwoods> and where does GRUB put the header?
04:27:37 <alberinfo_osdev_> i cant tell, i never searched for it, because before of this, my OS was just 32-bit, and then it worked fine, when i passed to long mode, i cant access the header anymore. can i see it with objdump?
04:29:37 <isaacwoods> doesn't GRUB give you its address in a register or something?
04:31:29 <alberinfo_osdev_> no..?but im seeing that eax and edi are set to 0x009000101FF00100
04:32:46 <isaacwoods> (I'm assuming this is GRUB2, not legacy) the physical address is in EBX
04:33:06 <isaacwoods> you need to map that address somewhere, and then use the virtual address you've mapped it to to access the header
04:33:48 <isaacwoods> as in it's in EBX when your code is first called, you need to make sure not to clobber it / put it somewhere else
04:35:01 <alberinfo_osdev_> ebx is 0x00001000
04:35:53 <isaacwoods> can you put your entry point code in a pastebin and post a link?
04:37:42 <alberinfo_osdev_> https://pastebin.com/bCmhD3nZ
04:41:02 <isaacwoods> have you checked to see if the header is at 0x1000 then (the xchg bx, bx suggests you've got a debugger handy)
04:48:06 <alberinfo_osdev_> yes
04:56:56 <isaacwoods> so then the header should be mapped to 0x1000 with your identity mapping - why are you accessing 0xc000066a
04:58:02 <alberinfo_osdev_> i have no idea. also, if i wait for the moment before calling kmain(when im in 64-bit long mode) and i push rbx, bochs says rbx = 0x108, instead of 0x1000
07:21:45 <alberinfo_osdev> hi, i have a question. i cant access the multiboot header, it throws a page fault, with cr2 set a little bit above the 3gb mark. the OS is working in 64-bit long mode, if i put it to work in pmode, it works just fine. what can i do? thanks in advance.
07:35:12 <geist> it's probably somethign to do with the way you're linking your binary
07:35:38 <geist> my guess is you've linked your binary to run at 0xc000.0000 (3GB) and grub istrying to load it there but it immediately faults
07:35:55 <geist> the multiboot header is almost always down <1MB
07:39:02 <alberinfo_osdev> yes, but the problem is that i dont link to 0xC0000000
07:40:35 <alberinfo_osdev> even, ebx is set to 0x10000
07:42:11 <geist> indeed, alas i dont have the time to be able to help you more
07:42:59 <alberinfo_osdev> ok ;(
08:29:23 <doug16k> alberinfo_osdev, can you show the output of objdump -p -h your-multiboot-executable
08:32:24 <alberinfo_osdev> yes.
08:33:48 <alberinfo_osdev> .text size: 13c, vma 0 lma 0 file off 280 align 2**4 contents, alloc, load, relo, readonly, code, .rodata: size: 40, vma 0
08:34:51 <alberinfo_osdev> lma 0, file off 3c0 2 **2, contents, alloc, laod, reloc, readonly, data. .bss; size: 13000, vma 0, lma 0, file off 400, 2**12, alloc
08:43:22 <doug16k> all vma 0 then?
08:43:28 <alberinfo_osdev> yes
08:43:33 <doug16k> that's bad
08:43:44 <alberinfo_osdev> ok, why?
08:44:04 <doug16k> code has to be linked at some address. how could it all be at zero?
08:44:49 <doug16k> can you show a paste of your linker script?
08:44:49 <alberinfo_osdev> well, ur right. but in my linker script i link it(in theory) to 1MB
08:44:55 <alberinfo_osdev> yes, wait a minute
08:46:26 <alberinfo_osdev> https://pastebin.com/5zb6dqXV
08:48:25 <doug16k> do you link with gcc or ld?
08:48:47 <alberinfo_osdev> ld
08:49:22 <doug16k> can you add this to the link command line and show paste of its output: -Map your-kernel-name.map
08:49:37 <doug16k> it will write a log of exactly where it placed each object file's things
08:49:49 <doug16k> will show what linker is really doing
08:54:59 <alberinfo_osdev> well, some things are in Spanish, due thats my native language. https://pastebin.com/4Szrs1jp
08:56:08 <doug16k> looks ok
08:56:21 <doug16k> you sure program headers were 0?
08:56:42 <doug16k> can you show paste of that objdump from earlier
08:56:49 <alberinfo_osdev> yes. if you want, i can directly copy the objdump to a pastebin
08:57:11 <doug16k> that will tell what a loader will do
08:57:31 <doug16k> link output looks fine, so next step is look at loader input
08:57:51 <doug16k> the -p part of the objdump (program headers)
08:58:30 <doug16k> the -h part lets you verify the correctness of the -p part
08:58:37 <doug16k> to some extent
08:59:58 <alberinfo_osdev> https://pastebin.com/081aNyyD
09:03:54 <doug16k> no -p output?
09:04:36 <doug16k> looks good though. your multiboot header is too far in though I think
09:05:50 <alberinfo_osdev> im sorry, i didnt notice i didnt pass -p. here's the link with -p option https://pastebin.com/f2nmm4ed
09:06:48 <doug16k> if you add this line inside .text output section it might help: . += SIZEOF_HEADERS;
09:07:09 <doug16k> the way you have it, it has to push text way out to 4KB boundary
09:07:26 <doug16k> if you reserve room for headers, it should tightly pack .text against the beginning of the executable
09:08:45 <doug16k> then your multiboot header should land at a lower file offset, increasing the probability of it working
09:09:26 <doug16k> throwing it all in .text isn't helping either I bet
09:09:41 <doug16k> why does it care so little?
09:10:09 <alberinfo_osdev> i dont know. it still without working, same cr2 addr.
09:10:29 <doug16k> there are plenty of linker heuristics that you will throw a wrench into if you have weird/careless permissions
09:10:50 <doug16k> you want rwx headers, code, data. :(
09:12:16 <doug16k> if you don't care if it is totally unprotected by page permissions you will probably have to say so explicitly with an explicit PHDRS declaration, or make your permissions reasonable and align data so executable readonly has a page boundary before no-execute readwrite
09:13:19 <alberinfo_osdev> ok. now i have .text just for .text, and .data is for .data and .rodata. still not working, how can i correct the permisons?
09:13:51 <doug16k> align rodata to 4KB and align .data to 4KB
09:14:12 <doug16k> put that . += SIZEOF_HEADERS; at the beginning of .text
09:14:37 <doug16k> this way would be fine: .rodata ALIGN(4K) : { *(.rodata*) }
09:14:42 <doug16k> same for .data
09:15:18 <doug16k> if you do that, then the magical things in the linker that infer program headers will be more apt to put things so .text is against offset 0
09:16:05 <alberinfo_osdev> ok. i aligned .data and .rodata to 4k, but still not working.
09:16:20 <doug16k> still file offset 0x1000 for .text?
09:16:30 <doug16k> the 1st program header I mean
09:17:21 <doug16k> you may need to explicitly add a PHDRS declaration and give FILEHDR PHDR in your first section
09:17:39 <doug16k> oops, FILEHDR PHDRS
09:18:14 <alberinfo_osdev> .text stills at ofset 0x280
09:19:01 <doug16k> good
09:19:06 <doug16k> that should work then
09:20:10 <doug16k> it was at 0x1000 before
09:20:31 <alberinfo_osdev> no, it doesnt. same errors as before. as a little adittion, 'cause i dont remeber if i said it. CR2 equals to RAX value
09:21:48 <doug16k> to see where it crashed, do this: x /6gx $rsp
09:22:04 <doug16k> <maybe error code> rip, cs, rflags, rsp, ss
09:22:26 <doug16k> that value for rip it pushed will be faulting instruction
09:22:46 <doug16k> I can predict that it was using rax as a pointer as in (%rax)
09:23:39 <doug16k> whether that works depends on exactly what exception occurred
09:24:01 <doug16k> if it died very ungracefully at stack overflow it is harder to backtrack
09:24:36 <doug16k> -d int qemu option should help in those pathological cases
09:26:38 <alberinfo_osdev> says 0x122fdc<bogus + 0>: 0x0 0x0 0x0 0x0, and 122fec <bogus + 16>: 0x0 0x0. RIP: 0x100206, cs 0x08, rflags 0x2, rsp 0x122fdc, ss 0x10. rax now is 0, when i try to acces to mboot_header is when cr2 is set and rax is with same value as cr2
09:44:33 <doug16k> do you know the address of your entry point?
09:44:39 <doug16k> or symbol name?
09:45:53 <doug16k> restart it, and put a linear breakpoint there with the lb command
09:46:06 <doug16k> continue it and when it hits the breakpoint, do: trace on
09:46:21 <doug16k> then continue, it will disassemble everything as it goes. you can see how it led up to that
09:46:33 <doug16k> to make the disassembly far more readable, you can build and load symbols
09:47:16 <doug16k> like this: https://github.com/doug65536/dgos/blob/master/gensymtab.sh
09:48:04 <doug16k> that takes an elf and generates a file suitable for the bochs ldsym command
09:48:28 <doug16k> it's a line delimited list of { address, name } pairs
09:49:12 <doug16k> then when it traces you can see what function it is and what global variable accesses are named
09:50:00 <doug16k> the trace becomes 1000x more readable when you see what functions it is calling and to where it is returning
09:55:34 <alberinfo_osdev> it just says Missing OBJDUMP variable, how can i pass objdump(and i think also $sort)?
09:58:00 <alberinfo_osdev> ah, i did it. wait a minute, ill give you a pastebin with the output.
09:59:07 <alberinfo_osdev> https://pastebin.com/FBgj75UE
10:11:25 <doug16k> see line 15 of the script
10:12:23 <doug16k> it takes 4 parameters: objdump executable to use, 'e', output filename, input filename
10:13:08 <doug16k> so you might run ./gensymtab.sh objdump e something.sym your-kernel-exe
10:13:28 <doug16k> objdump is there so you can pass cross compiler objdump, like x86_64-elf-objdump
10:13:56 <doug16k> or whatever configure step used
10:14:35 <doug16k> my other thing passes 'e' for the type. I forget why
10:14:57 <doug16k> might be obsolete thing
10:17:16 <alberinfo_osdev> well, still saying missing objdump variable. is the pastebin the thing you want?
10:28:38 <doug16k> I was trying to get you symbols to load in bochs
10:28:45 <doug16k> that generates a symbol file
10:29:01 <doug16k> then when you trace it you can see what it is doing more clearly
10:29:52 <doug16k> that script takes the objdump output and extracts the name and address of the right lines and produces a symbol file that bochs can load
10:30:51 <remexre> if I'm building a PC as an osdev target platform, does mobo manufacturer matter at all?
10:32:14 <doug16k> remexre, if you get one with intel lan you will have no trouble getting docs to make a nic driver
10:34:26 <remexre> doug16k: okay, so like an ASRock H110M-ITX ?
10:38:29 <doug16k> I stick to asus usually
10:38:54 <doug16k> if you don't find one with intel lan, then add a modest intel nic to the build
10:39:44 <remexre> ok
10:41:05 <doug16k> I went and got the datasheet and programming stuff before buying. I got an I350-T4 quad gigabit nic
10:41:38 <doug16k> comparable or not-quad one won't cost too much
10:42:32 <doug16k> that nic has lots of offloads, so you can fit that into your nic driver API somehow sometime
10:42:37 <alberinfo_osdev> doug, it still not working. as you didnt response, ill ask again, did you see this pastebin? https://pastebin.com/FBgj75UE
10:42:48 <doug16k> yes
10:42:53 <doug16k> that is the objdump output right?
10:43:01 <alberinfo_osdev> yes
10:43:09 <doug16k> that whole script is what transforms that to bochs symbols
10:43:54 <doug16k> if you run that script with 4 parameters it will generate a bochs symbol file you can load in bochs with the ldsym command
10:44:35 <doug16k> the 4 parameters are, 1) the objdump executable path, 2) the letter 'e', 3) output filename (your sym file), 4) input (your multiboot exe)
10:45:13 <doug16k> oh there's a bug that I didn't notice because I define objdump
10:45:28 <doug16k> make those $OBJDUMP lowercase. sorry
10:46:15 <doug16k> should be using the objdump assigned on line 15 when it runs 23 or 31
10:48:08 <alberinfo_osdev> well, i lowercased all the $objdump, and calling the file with ./gensymtab.sh objdump e kern.sym bootloader.o, and the output is still being: missing objdump variable.
10:48:17 <doug16k> ah, it needs SORT defined too. add SORT=$(which sort) early
10:48:29 <doug16k> or just SORT=sort
10:48:54 <doug16k> this is run by make so it benefits from some exported things
10:49:18 <doug16k> when you bare run it you need to specify SORT
10:49:57 <doug16k> you can specify those things by putting the assignment before the command: SORT=sort ./gensymtab.sh objdump e kernel.sym kernel
10:50:30 <doug16k> it will put SORT=sort in the program's initial environment
10:50:37 <doug16k> it won't affect this shell's environment
10:51:26 <doug16k> kernel.sym should look like a bare list of address name pairs
10:52:14 <doug16k> (the output file)
10:52:19 <alberinfo_osdev> calling it with SORT=sort ./gensymtab objdump e kern.sym bootloader.o says the same, i dont know if im doing something wrong or its your code
10:52:40 <doug16k> what does "the same" mean?
10:52:47 <doug16k> I forget what it said exactly
10:52:59 <alberinfo_osdev> missing objdump variable
10:53:07 <doug16k> you made those OBJDUMP lowercase?
10:53:13 <alberinfo_osdev> yes
10:53:22 <doug16k> then how can OBJDUMP be missing
10:53:28 <alberinfo_osdev> i dont know
10:53:35 <doug16k> operator error obviously
10:54:00 <doug16k> didn't save, wrong file, missed one, could be lots of legitimate reasons
10:54:53 <doug16k> bash variables are case sensitive
10:55:49 <doug16k> does it say missing objdump or missing OBJDUMP?
10:56:06 <alberinfo_osdev> missing OBJDUMP
10:56:21 <doug16k> changed line 3?
10:56:36 <doug16k> ah!
10:56:41 <doug16k> remove that whole if on line 3
10:56:57 <doug16k> remove 3 thru 8
10:57:35 <doug16k> sorry that was sanity checking build variables
10:58:08 <doug16k> you can remove 2nd if too and just change the $SORT to sort
10:58:49 <alberinfo_osdev> and now it works ;) it ended up with i think are addresses, and at its right, haves what it corresponds to(i think)
10:59:14 <doug16k> now in bochs, ldsym global path-to-that-file-here
10:59:27 <doug16k> now when you step, trace, etc, it will resolve symbol names
11:00:05 <doug16k> I actually contributed a fair bit of the symbolic disassembly behaviour and enabled x86_64 a while back
11:00:20 <doug16k> the debugger was broken as hell in bochs on x86_64 before
11:02:46 <doug16k> enabled x86_64 symbols I meant. it was truncating them to 32 bit
11:03:32 <doug16k> it didn't even decipher rip relative for you. you had to know x86 down to machine language level and decode where it referred
11:03:33 <alberinfo_osdev> ok, i started bochs, i putted in the bochsrc debug_symbols: file=kern.sym, and now i think its loading the symbols
11:03:59 <doug16k> nice
11:04:15 <doug16k> disassembly looks way more readable?
11:04:30 <remexre> doug16k: is e.g. http://www.intel.com/content/www/us/en/embedded/products/networking/ethernet-connection-i219-datasheet.html?asset=10141 the sort of datasheet you're talking about?
11:04:30 <alberinfo_osdev> god hell it does :)
11:04:32 <doug16k> call calls a thing with a name, not some hex random digits?
11:05:29 <alberinfo_osdev> yes, for example when i set up data segments for lmode, says lmode_start+5
11:06:09 <doug16k> alberinfo_osdev, \o/
11:06:27 <doug16k> should be a lot easier now, if you do trace on, and continue until a crash, you can follow what it did
11:07:01 <doug16k> I'd set breakpoint at entry point, then continue, then trace on once in your code, then continue and trace that
11:07:10 <doug16k> it will be bazillions of bios insns otherwise
11:09:27 <doug16k> remexre, that's not really at the right level of abstraction I don't think. looks like a PHY chip
11:09:51 <doug16k> smbus to pcie? D:
11:10:00 <remexre> yeah, I thought it seemed too EE
11:10:56 <doug16k> need the comparable NIC datasheet. not chip-on-the-nic datasheet preferably :)
11:11:29 <doug16k> if you knew what the driver says when it is installed you can infer approximately which discrete card that is and go from there
11:11:45 <doug16k> hard when picking one though :(
11:12:05 <doug16k> or you can not worry too much about the mobo nic and just get a plug in pcie nic
11:12:37 <remexre> The mobo mfr's page says it's an I219-V, and I've got a box w/ one of those rn
11:12:55 <remexre> it says "Kernel driver in use: e1000e" in lspci -vvv, is that what you mean?
11:13:18 <doug16k> unfortunately "e1000" is intel's universal driver thing
11:13:24 <remexre> damn
11:13:42 <doug16k> if they made a piece of break with an RJ-45 jack and PCIe edge connector, e1000 would be the driver
11:13:48 <doug16k> bread*
11:13:55 <remexre> lol
11:15:44 <alberinfo_osdev> well, says it crashed on kmain+1a, which beings to be when i try to print bootloader name
11:16:43 <doug16k> alberinfo_osdev, nice. knowing where should help
11:17:07 <alberinfo_osdev> yes, but still, i cant access the mboot_header, and i dont know why
11:17:18 <doug16k> paging is on?
11:17:35 <alberinfo_osdev> yes, if not, it wont be in long mode
11:17:41 <doug16k> right long mode
11:17:52 <doug16k> does info tab work?
11:17:57 <alberinfo_osdev> yes
11:18:11 <doug16k> what address is mboot_header supposed to be at?
11:18:27 <alberinfo_osdev> below 1mb
11:19:01 <doug16k> what do you mean by "can't access"
11:19:04 <doug16k> crashes?
11:19:54 <doug16k> or do you mean when you access it, you get nonsense that can't be right?
11:21:28 <alberinfo_osdev> when i access mboot_header, cr2 is set, with the same value as RAX, if you remember, is the same error as the start
11:21:59 <doug16k> rax is the address of the mboot header/
11:22:00 <doug16k> ?
11:22:14 <alberinfo_osdev> no, is C000066A
11:23:09 <doug16k> cr2 is set when there is a page fault
11:23:32 <doug16k> what does info tlb say for 00000000c0000000
11:23:52 <remexre> doug16k: all roads I can find seem to be leading back to that same datasheet :(
11:25:42 <alberinfo_osdev> nothing, because i just map till 4mb
11:26:30 <doug16k> remexre, this is what my sheet looks like: https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/ethernet-controller-i350-datasheet.pdf
11:26:45 <doug16k> notice chapter4 thru 8
11:28:07 <doug16k> if possible get a nic where you can get docs something like that
11:30:06 <remexre> doug16k: okay, thanks
11:31:16 <alberinfo_osdev> hey doug, if i do xp 0xC0000000, says it's value is 0xFFFFFFFF, but i never got touching even further than the 4mb mapped
11:34:08 <alberinfo_osdev> more information, it crashes when calling strlength, but wtf, i used the same before and gave me no problems
11:34:27 <doug16k> that proves nothing
11:34:53 <doug16k> working in one case does not prove that it is perfect
11:35:10 <doug16k> is it compiled code?
11:35:15 <doug16k> is the stack correctly aligned?
11:35:41 <doug16k> red-zone off?
11:35:51 <alberinfo_osdev> its compiled code, and the stack is correctly aligned.
11:35:56 <alberinfo_osdev> whats red-zone off?
11:36:16 <doug16k> on x86-64 there is an area of the stack below the stack pointer called the "red zone:
11:36:42 <zid> It's where the monsters from the dungeon dimensions break through the thin vaneer we call reality
11:36:50 <doug16k> if you are a leaf function and you use less than 128 bytes of stack for variables it can skip updating the stack pointer and put stuff on the wrong side of the stack pointer
11:37:17 <doug16k> that can't work in kernel code - it relies on the architecture switching stack on interrupt
11:37:24 <doug16k> it works in user code
11:37:40 <doug16k> default is to have that on. you need -mno-red-zone
11:38:23 <doug16k> if red zone is off in kernel code you will eventually get extremely weird problems with things being clobbered on interrupt
11:38:36 <doug16k> if on you get problems I mean
11:39:30 <alberinfo_osdev> ok... with -mno-red-zone stills not working, if i reserve more space for the stack, could it work?
11:41:03 <alberinfo_osdev> ok, it didnt. i reserve 64KiB, reserving 1MiB did not work
11:41:22 <doug16k> 32KB is quite a bit
11:41:50 <alberinfo_osdev> so 64KB is way too low?
11:42:04 <doug16k> KB
11:42:22 <doug16k> 32KB is plenty, a good size for kernel stack
11:42:49 <doug16k> assuming you don't abuse the stack with giant local arrays
11:43:52 <doug16k> I use 32KB and I never hit an overflow that wasn't an infinite recursion
11:44:13 <alberinfo_osdev> no, the biggest array i have is one for printing characters with VGA graphical modes, but the char array is not even completed
11:44:47 <adu> is there a room for bios programming?
11:45:09 <doug16k> seabios has a room
11:45:42 <doug16k> if not here try OFTC
11:46:30 <adu> I've been fascinated by the ring (-2), (-3), amt, txe, me, stuff
11:47:41 <zid> One day, I will get around to writing in-bios no-ram tetris
11:47:46 <zid> when I have two machines set up
11:48:05 <zid> Run it out of cache, load resources from rom
11:52:47 <alberinfo_osdev> doug_16k, if i print the bootloader_name in hex, and then try to pass it to str, i see its wrong, so - 1)Multiboot header is wrong, and 2)if i try to directly print the value, it page faults, but when i pass directly a string, it does not. which seem to lead me to be that mapping is wron, but it isn't. info tab on bochs, and info mem in qemu says the're fine.
11:53:35 <adu> zid: you should make it run on sparc
11:55:47 <zid> can sparc run with no ram, are you going to send me a sparc machine
11:56:21 <adu> zid: Intel's ICH11 is reportedly backed by an on-board sparc cpu
11:56:45 <zid> I have ICH11 and ICH10R :P
11:56:51 <adu> or was it ICH10... don't remember
11:57:01 <zid> I don't fancy messing around with ME though
11:57:05 <adu> zid: you should do a BIOS dump and disassbmle it
11:57:14 <zid> I've already disassembled my bios
11:57:19 <zid> I had to add support for my cpu
11:57:20 <adu> or wait, not bios, SPI flash dump
11:58:21 <adu> SPI flash memory has volumes: FD, BIOS, ME, GBE
11:59:33 <adu> I've been learning all about it from coreboot videos on youtube