channel logs for 2004 - 2010 are archived at http://tunes.org/~nef/logs/old/ ·· can't be searched
#osdev2 = #osdev @ Libera from 23may2021 to present
#osdev @ OPN/FreeNode from 3apr2001 to 23may2021
all other channels are on OPN/FreeNode from 2004 to present
http://bespin.org/~qz/search/?view=1&c=osdev2&y=24&m=7&d=13
05:32:00 <xenos1984> Maybe they had a pointer to a variable in some other register than rsp, which uses ds by default, so that it was `mov [some other register], eax` instead?
06:37:00 <chiselfuse> xenos1984: oh i see why it was 'stack' instead of 'rsp` then hmm
06:40:00 <chiselfuse> well either way, i checked a process in linux and: `ds` is 0, `ss` is 0x2b (entry 2 in gdt, RPL=3). here `ds` and `ss` are different though they must both have 0 as base address. i assume he must've been talking about x86
06:40:00 <zid> cpu doesn't care if you don't fill out ss in long mode for some reason
06:41:00 <zid> err ds? whatever
06:41:00 <zid> 2b i s not entry 2 last I checked though?
06:42:00 <zid> (5*8) + 3 = 0x2B
06:42:00 <chiselfuse> i don't understand why in long mode one data segment (ss) gets filled to point to a descriptor that contains a bunch of attributes such as permissions while still having 0 base address while all other data segments (es/ds) are just nulled out
06:42:00 <zid> cpu doesn't bother to check them, and es is unused
06:43:00 <zid> not sure *why* it doesn't check them, but it is a thing that is true
06:43:00 <chiselfuse> zid: why does it bother with ss then
06:43:00 <zid> the only thing the tss has it in it is ss and rsp
06:43:00 <zid> so that it can do a correct context switch
06:43:00 <zid> it just assumes ds from ss
06:43:00 <zid> and cs
06:44:00 <zid> because.. long mode isn't segmented
06:45:00 <zid> it's literally just filling in basically dummy values, for the 1 or 2 bits of actually mutable pieces of data all long mode cares about from all that segmentation stuff. Which if segmentation had not existed, would certainly be stored somewhere else entirely.
06:45:00 <chiselfuse> ah yea you're correct, it's index 5, not 2
06:45:00 <zid> for interrupts to happen it needs to have a ring0 ss:rsp to push to
06:46:00 <zid> so that's pretty much the only valid field left in the tss that long mode actually cares about
06:50:00 <zid> right got interrupted myself, ironic. x
06:50:00 <zid> Which means, ss is the only selector that really ever changes and gets reloaded
06:50:00 <zid> so ds can just hang out being broken with no repercussions
12:39:00 <adder> I'm wondering if I should disable paging before I map the available ranges... I'm currently not, and I'm getting a pagefault during the mapping process. I'm using limine
13:05:00 <heat> i can't advise on limine, sorry
13:10:00 <adder> heat, what do you use?
13:11:00 <adder> handrolled I bet :)
13:11:00 <heat> grub
13:12:00 <adder> I think I might start over again
13:13:00 <mjg> heat did a bunch of text replacement on lfs
13:13:00 <adder> I don't like this thing a tiny bit
13:13:00 <mjg> s/Linux/Onyx/
13:13:00 <mjg> now claims he has a kernel
13:14:00 <netbsduser> adder: limine already does map the available memory ranges into the higher half already
13:15:00 <Ermine> heat: do your pipewire issues trigger when you use bluetooth headphones or when you use wired headphones as well?
13:15:00 <adder> netbsduser: so I'm not sure what part of paging am I supposed to do
13:15:00 <adder> do I need to set up page tables and reload cr3?
13:16:00 <adder> do I need to iterate over the memory maps and map that space then?
13:17:00 <netbsduser> i just reuse the tables limine sets up initially, but that does mean i don't reclaim the memory marked bootloader-reclaimable
13:17:00 <netbsduser> it is up to you what you want to do
13:17:00 <netbsduser> you can do likewise, or you could assemble your own page tables such that you map the kernel and map as much or as little of the available memory as you want wherever you want
13:18:00 <adder> I am trying to set up my own tables and reload cr3 but am pagefaulting in the process
13:19:00 <netbsduser> then a translation is failing - the tables do not map everything you need them to map, or maybe they are bogus
14:17:00 <heat> Ermine, wired. bluetooth headphones in linux lol
14:19:00 <Ermine> well, in my case pw shits itself only with bluetooth ones, while wired work okay
14:44:00 <heat> page table reclamation at unmap time, yes or no?
14:45:00 <gog> no
14:45:00 <mjg> yes
14:45:00 * mjg <-- contrarian
14:45:00 * gog claw mjg
14:46:00 * mjg hisses at gog
14:46:00 <gog> :o
14:47:00 <heat> mjg are you aware of the freebsd behavior on that
14:47:00 <Matt|home> morn.
14:47:00 <mjg> no
14:48:00 <mjg> it's probably stupid, so just find out what it is and do the opposite
14:48:00 <mjg> yer welcome
14:48:00 <heat> linox only does it on munmap but it's a terrible approach where they go down the page tablen (and unmap pagen), then go back down later to free them
14:49:00 <heat> truly next level wtf ideas
14:49:00 <Matt|home> off-topic, brain no worky - average wifi connection speed w home router in MB/s ?
14:49:00 <mjg> they could stash them
14:49:00 <mjg> no need to do the walk twice
14:49:00 <Matt|home> 3/5g
14:49:00 <Matt|home> im guessing 25-30
14:50:00 <mjg> this is a rather bad channel to ask this mon
14:50:00 <heat> mjg, could also just free them at the same time they unmap
14:50:00 <mjg> i presume there are locks held while processing it
14:50:00 <mjg> so liek don't artificially extend hold time kthx
14:51:00 <mjg> their free is dog slow
14:51:00 <heat> they'll hold the locks again to free the page tables, i hope at least
14:51:00 <mjg> look mon i don't know the original reasoning
14:51:00 <mjg> i'm saying you can probably collect tehse tables and free them after
14:51:00 <heat> YOU KNOW NOTHING
14:51:00 <Matt|home> o.o
14:51:00 <mjg> harsh but fair
14:51:00 <heat> the annoying big issue when doing this kind of stuff is that it's hard to know if the page table is completely free
14:52:00 <heat> so, uh, stashing a page table refcount in a struct page or whatever? but then every set pte operation turns into some kind of cmpxchg
14:52:00 <heat> it might not be worth optimizing
14:53:00 <Matt|home> yay i got samba set up and working.. my laptop is no longer e-waste, i can use it as a really shitty file server :D
15:08:00 <vin> madvise seem to just be an "advise" to the kernel about what to do with a page. For example, when I say MADV_DONTNEED on a set of pages, even after waiting for a few seconds I see the RSS of that application not decreasing. Is there a more aggresive way to hint the kernel?
15:11:00 <heat> MADV_DONTNEED is not an advice
15:11:00 <heat> you're probably holding it wrong
15:12:00 <vin> I see! I will double check if my start address is page aligned and the size is smaller/equal to the total mmap size
15:16:00 <heat> i know RSS has some counter fuckyness, it's possible you're seeing old values. or the pages you think are present actually aren't
15:21:00 <vin> is there a more reliable way to tell if the pages are swapped out? If not by looking at RSS
15:23:00 <vin> Also is there a source that documetns the unreliability of RSS somewhere?
16:02:00 <clever> vin: on linux, /proc/$PID/pagemap is basically an export of the leaves of the paging table
16:02:00 <clever> for a 64bit system, it is a flat array of 64bit physical addresses, telling you what page every page of the VM maps to
16:03:00 <clever> and with 64bit virtual mode, that makes it an array of uint64_t[2^64]
16:03:00 <clever> you can just read any offset you want to query, and if its resident, youll get a non-zero number
16:08:00 <vin> Perfect, thanks clever! This is almost like page table scanning (for access bits) in user space.
17:15:00 <vin> Seems like I can't use that clever from the madvise man page: On MADV_DONTNEED "The kernel is free to delay freeing the pages until an appropriate moment. The resident set size (RSS) of the calling process will be immediately reduced however." So the pagemap might still show a page as resident even if DONTNEED is called on it
17:16:00 <vin> Only the RSS seem to be reduced right away. Unfortunately though my RSS remains the same after DONTNEED
17:59:00 <heat> vin, that is NOT what the man page says
17:59:00 <heat> man page says: ""The kernel is free to delay freeing the pages until an appropriate moment. The resident set size (RSS) of the calling process will be immediately reduced however."
17:59:00 <heat> so, simply put: the pages may not be freed, but the rss is reduced and the pages are unmapped
18:00:00 <heat> (pages are only not freed when e.g they're shared)
18:00:00 <vin> meaning the offset of /proc/self/pagemap will be 0 even if the pages aren't freed yet?
18:03:00 <heat> yes
18:03:00 <heat> not offset, value i imagine
18:09:00 <vin> Got it!
18:11:00 <heat> re: RSS inaccuracy
18:11:00 <heat> https://man7.org/linux/man-pages/man5/proc_pid_statm.5.html
18:11:00 <bslsk05> man7.org: proc_pid_statm(5) - Linux manual page
18:11:00 <heat> "Some of these values are inaccurate because of a kernel-
18:11:00 <heat> internal scalability optimization. If accurate values are
18:11:00 <heat> required, use /proc/pid/smaps or /proc/pid/smaps_rollup
18:11:00 <heat> detailed information."
18:11:00 <heat> instead, which are much slower but provide accurate,
18:11:00 <heat> TL;DR mjg fucked it up
18:12:00 <mjg> i literally did not, but probably would have
18:13:00 <heat> didn't you touch rss a few weeks back or so?
18:13:00 <mjg> i did not *alter* the state mofo
18:13:00 <heat> damn. mofer.
18:13:00 <mjg> that was all g men
18:13:00 <mjg> i only made it suffer less on fork/exec
18:15:00 <mjg> and have some more patchen in the makingz :X
18:15:00 <mjg> however
18:15:00 <mjg> i don't see any reasons for that statm sucker to even know about it
18:15:00 <mjg> it literally could sum it up if needed
18:16:00 <heat> yeah ONYX USES AN ATOMIC MON
18:16:00 <mjg> wll
18:16:00 <mjg> linux > onyx then
18:16:00 <mjg> :X
18:16:00 <heat> so no idea what kind of games they're pulling off with rss
18:16:00 <mjg> it's a per-cpu counter with batching
18:16:00 <mjg> 's like the most cliche game in town
18:17:00 <mjg> the counter used to be distributed per-thread
18:19:00 <mjg> so if anything it was those guys who fucked it up 8S
18:19:00 <heat> aha yes a classic
18:19:00 <heat> was the terribleness the pcpu allocator?
18:19:00 <mjg> yes
18:19:00 <mjg> i have further patchen to sort it out
18:19:00 <mjg> did not productize them yet
18:20:00 <heat> productize? is corporate-speak mjg upon us
18:20:00 <mjg> 's ok word mofo
18:20:00 <mjg> what i did not implement yet but will have to add is the destructor support for slabz
18:20:00 <mjg> to pair up with ctor
18:21:00 <heat> haha good luck
18:21:00 <mjg> ye we will see
18:21:00 <mjg> i have an idea how to sort things out without adding one, but it would be very hacky
18:22:00 <vin> the man page says looks at smaps or smaps_rollup and not pagemap for accurate rss estimation. Why is that?
18:22:00 <heat> i believe smaps does proper page table walking
18:24:00 <heat> yep, code checks out
18:25:00 <heat> well, pagemap would also accurately estimate RSS probably, but without locks
18:25:00 <heat> but pagemap really isn't the way to estimate RSS :)
18:25:00 <mjg> my estimate: 20
18:27:00 <chiselfuse> https://github.com/skeeto/pure-linux-threads-demo/blob/master/threads-x86_64.s
18:27:00 <bslsk05> github.com: pure-linux-threads-demo/threads-x86_64.s at master · skeeto/pure-linux-threads-demo · GitHub
18:27:00 <heat> my early onyx processen actually hover around ~1MB RSS
18:27:00 <heat> probably because of being dynamically linked
18:27:00 <chiselfuse> when assembled, how does this exit? i imagine execution starts at _start, but doesn't it continue after the last instruction in _start? where does it stop?
18:28:00 <heat> it looks forever
18:28:00 <heat> loops
18:28:00 <chiselfuse> oh, right it's an unconditional jump
18:29:00 <chiselfuse> wait, weird, i built and ran it and it stops after a while
18:30:00 <mjg> some overflow ?
18:31:00 <chiselfuse> doesn't say it segfaultetd
18:31:00 <heat> check_count does exit actually
18:31:00 <chiselfuse> oh right
18:43:00 <mjg> __ia32_sys_vfork+72
18:43:00 <mjg> do_syscall_64+82
18:43:00 <mjg> backtrace
18:46:00 <heat> great backtrace mon
18:46:00 <mjg> thanks
19:18:00 * Ermine yawns
19:44:00 <chiselfuse> apparently you can _not_ share signal handlers across threads in linux with CLONE_SIGHAND
20:42:00 <chiselfuse> at the start of a c program i compiled, the value at fs:28h is put onto the stack of main never to be accessed again (i checked this by adding a tracepoint). what is it for?
20:43:00 <zid> reminds me of a stack canary
20:43:00 <zid> but you said it's not checked
20:43:00 <zid> so.. boot.s is written with them but the biary isn't built with it? idk
20:44:00 <chiselfuse> zid: how are stack canaries checked?
20:44:00 <zid> one of the -fstack-protector= options will check them
20:46:00 <chiselfuse> meh, 0x0.st replies with the text "segmentation fault" if you try to post an elf to it
20:46:00 <chiselfuse> v funny
20:47:00 <chiselfuse> https://0x0.st/XLL8.c
20:47:00 <chiselfuse> just `gcc main.c`
20:48:00 <zid> I already said what my guess was, you're seeing the canary *once*, because boot.S (from the libc startup code) is initializing it, but the actual body of the program isn't built to bother checking it
20:48:00 <chiselfuse> https://0x0.st/XLLN.txt
20:48:00 <zid> because the startup code doesn't know if stack protectors iwll be used or not
20:48:00 <chiselfuse> oh that message was for me
20:48:00 * kof673 .oO ( uuencode )
20:48:00 <zid> so is just blindly doing it
20:48:00 <chiselfuse> i see
20:49:00 * kof673 .oO ( boo )
21:44:00 <heat> mjg, friend: https://thundergolfer.com/computers-are-fast
21:44:00 <bslsk05> thundergolfer.com: Computers are fast – Jonathon Belotti [thundergolfer]
21:45:00 <heat> its a hard quiz i'll tell you that
21:53:00 <nikolar> I was never off more than one away from greens
21:55:00 <chiselfuse> does daily driving a computer that is 15 years old make you not know computers automatically then
21:57:00 <heat> >A newer computer won’t make any of the benchmark code run 1000x faster.
21:57:00 <nikolar> Correct
21:57:00 <nikolar> 15 years ago was 2009
22:08:00 <zid> 2011 is the minimum viable compuer
22:08:00 <nikolar> Lel
22:10:00 <zid> man, you REALLY gotta know python for that part
22:10:00 <zid> how the fuck should I know if their optimizer elides empty loops, or if it ends up interpreted and thus 10/s
22:13:00 <zid> Also, this guy's computer has really shit memory bw
22:13:00 <zid> guess because it's a laptop
22:16:00 <zid> https://level1techs.us-east-1.linodeobjects.com/optimized/4X/0/e/9/0e9be5cabc250c0f40fee30984b99f7d4705845b_2_856x825.jpeg Imagine having more memory latency than a zx spectrum though
22:16:00 <nikolar> zid but it's a new arm apple
22:16:00 <nikolar> *macbook
22:17:00 <zid> yes, a macbook is a laptop
22:17:00 <zid> all laptops have shit dram, I should have realized
22:43:00 <zid> nikolapdp: https://www.youtube.com/watch?v=mVeqQkdM9_4
22:43:00 <bslsk05> 'Why You Shouldn't Learn Romanian' by Living Ironically in Europe (00:19:45)
22:44:00 <nikolar> Heh great title
22:49:00 <zid> romanul vrea sa plece din romania cat mai repede
22:50:00 <zid> romanul vor sa fure un radio
22:55:00 <nikolar> Ah, you're confusing them with gypsies
22:58:00 <zid> no? gypsies aren't plumbers
23:09:00 <kof673> supposedly :D > Kheb-si the child or descendant of Kheb, Kab, Kab-t, Egypt, is our Gipsy > the Lee tribe of Gipsies, who told Barrow that their name in Egypt signified an onion > Rekh (Eg.) the equivalent of Leek, signifies heat, and Leek modifies into Lee. Moreover, the onion is worn in the hat by the Welsh, and the Egyptian Hut is both onion and hat.
23:09:00 <kof673> it was the style at the time ...
23:20:00 <heat> i've realized i have a bunch of tricky page table clearing logic i haven't implemented
23:20:00 <heat> annoyin
23:20:00 <heat> (mainly when it comes to invalidating the pt walker cache, *especially* before freeing the page table)