Apart from binary just being binary intrinsically (decimal notation, etc.) the binary sequences still have to be programmed to mean & initiate certain tasks. With that being said, & to be more specific/clarify:
How does one implement a custom assembly language with custom mnemonics starting from scratch, defining the values & definitions for the binary to execute specific tasks at specific address registers? (i.e. mapping keyboard keys, etc. then onto the assembly language, assembler, etc). (for implementation on a 64 bit computer).
In a nutshell, I'm inquiring about starting from scratch from binary. Essentially the lowest level of operations. Is a custom CPU/GPU necessary for this kind of implementation?
So you ask about assembly language which is just a low level programming language often intended to have a direct relationship with a specific instruction set architecture (ISA or instruction set). If you are asking if you can make up an new instruction set for an existing ISA, absolutely nothing is stopping you from doing that instead of something like
mov ax,[bx]
you could make that
ldr ax,[bx]
or
lw ax,(bx)
or
bob pickle,(pencil)
It is your choice, then simply write a parser and generate the instructions.
Done.
Erik pretty much covered it if the question is about creating a new instruction set from scratch then from that one or more assembly languages.
You first need to create the instruction set and honestly due to the question(s) you asked you are not familiar with instruction sets nor assembly language. So today we have the advantage of being able to examine many many instruction sets and assembly languages and get comfortable with them. In the same way if you want to build a house this is not caveman times you do not have to figure this out from scratch, there are millions/billions of homes to go look at and not just the basics of walls and a roof, but do I like the garage up front or in back or none, one level or multi-story. Basement, etc. Same here, what do I like about instruction sets what do I not like. There needs to be a strong reason to want to create a new one so you already have to have something you do not like or some business or educational reason, which means you have details that you wish to fix or implement that other instruction sets do not have or one does and another doesn't and you want to find a middle ground.
Let us forget about patents for now which will interfere with any real implementation.
You have to decide things like fixed length or variable length, but long before you get here you already have experience with many instruction sets and you have in your mind the reason why you are trying to create a new one vs just using one, or making a clone (again forgetting about legal problems).
For a general purpose processor you will need the basics, a number of alu operations add, sub, and, xor, etc...You will need some basic load and store instructions. Some form of unconditional and conditional branching, and so on.
Are you making a stack based processor or a normal style one?
If in your mind before you started this you wanted variable length you still need a minimum size so maybe like x86 you want 8 bit instructions and that opcode will either be the whole instruction or describe that more bytes are required. Or maybe you want 16 bit and still allow for variable length or go with fixed length which probably means 32 bits or 64 bits, but you can see with arm, mips, risc-v and probably others that you can have 16 bit instructions and 32 bit (and larger) with some scheme to switch between them.
And then you bang out the encoding. Things that are pc-relative you want to probably have a lot of immediate bits, the more bits the longer it can reach, assuming fixed length instructions, if variable length then you still have to decide if there is a limit or not, or a scheme to pick the offset size in the encoding. If fixed then you need a way to indicate which bits cover these large immediate encodings.
So while ARM's encoding seems chaotic to folks that cut their teeth on MIPS you can go back to the Acorn diagrams or even parts of the ARM ARM, and see that it is not chaotic at all, starting with the top bits instructions that want more immediate bits can be determined with fewer bits, then they add another bit and another bit to the encoding making unique starting bits and working through instructions that need fewer bits to determine the instruction down to other instructions that can tolerate more. For example 1/4th of the instruction set is dedicated to one type of instructions, then 1/8th is dedicated to another type, 1/16th of the possible encodings another flavor and so on. Another approach is mips like where you have some bits reserved as opcode bits (and then for some opcodes other bits then consumed as extended opcode bits) but being overall limited by what you can fit in any instruction, but the decoding is much easier, a tradeoff.
You can be CISC like and assume a microcoded architecture, a simple 8 bit opcode where the bits do not indicate anything they simply form a unique number that is looked up in a table to figure out what it does. You can look at the visual6502 page and instruction sets from that generation, the opcode basically is an address into a rom, the rom contains the microcode of the steps to implement that instruction.
You can go for a vliw instruction set, which is kind of like exposed microcode, you can pack that thing with control signals and register file addresses, whatever. And then have state machines that in a more simplified manner than even RISC decode and execute.
You have numerous other basic instruction set problems are there registers or is it stack based, is there one register, two or 8 or 16 or 256. What are your addressing modes, what does the processor bus look like, etc.
Eventually you start to settle in on an instruction set. Naturally as you develop this you are using names for instructions in your head or written down, this one loads from memory using a register contents as the address. This one adds two things. This one copies one register contents to another. So as part of keeping track for yourself or communicating with others, you have these terms, and your assembly language will probably come from that. And then you get into is it a copy or is it a move do you spell it mov or move. Do you use mov or load and store and do you spell them ld, load, ldr, etc. Do you like the destination first, do you like destination last. Do you have immediates/constants (add register 7 = register 3 + 5) if so do you need a parsing marker to indicate these and do you default to decimal or hex or octal or binary as the encoding of the values in the syntax.
So then you develop an assembly language and then want to develop a tool. Do you do it brute force or do you do some bison/flex thing? Or other solution?
Do you allow the assembler to make complete binaries with .org like statements or do you also or only force objects and then have to create a linker to link the objects into a binary? And if you do that then you can create other languages and you can work on that later.
You will want the assembler at a minimum pretty quick before you get too deep into the implementation so that you can work on testing the processor without having to hand code too much machine code.
There are and have been instruction sets that have things like tty instructions and others that touch specific peripherals in the processor. But in general the peripherals are mapped with some address on a bus (I/O bus or memory bus or a combination) and the instruction set does not care, in the long run that provides a more flexible and the ability to add or remove peripherals as the world evolves without having to make changes to the instruction set. You probably do not want to integrate peripherals in general into the instruction set.
Now the gpu which is often a completely separate instruction set that goes through the same process but the goal is tuned more for a specific application than a general purpose processor. But it is certainly possible to combine these have an instruction set that has both general processing features and also has graphics processing features. You might find you can figure out a way to have this run well, but often this does not help, GPUs evolved to offload work from the main processor so the main processor can keep working as hard as it can and passes off specific work to the gpu. Please draw me a square rather than here are the 4000 pixels I have to compute one at a time.
Then you need compilers and then operating system(s), applications and then customers. And as you can see with say wintel vs linux on arm, one is just as useful to a user as another, one consumes significantly more power, etc. But for reasons that are not technical you will have a hard time breaking the momentum of the existing world.
So bottom line the fact you are asking this question means you are not ready to start a task like this, you need to go study at least a dozen instruction sets, and their assembly languages, ideally from different companies from different periods in history. pdp8, pdp11, 6502, 8051, 68K, x86, arm, risc-v, PIC, msp430, avr mips, powerpc, sparc, just to name a few. And possibly some others like what was it the amd29000, and a stack based one, zpu? Or maybe just do this, look at every one of the instruction sets that gnu supports and all the ones that llvm supports. You will keep the pdp11 which should be on your list, period, but lose pdp8 and 6502 which have some unique features that may or may not be apparent on initial inspection but are worth knowing to keep your mind open. Then for the gpu side just go study some gpus, the one in the raspberry pi is somewhat documented now and there are no doubt others.
If you are serious about this and are considering a product that can compete with current products you need to be willing to invest tens to a hundred or so millions of dollars for the first chip/processor. And that may get you something that runs in the single ghz range. Multiply that by four if you want to get to the next stage may be 2ghz plus. Sure you can build a simple wee tiny thing that does almost nothing (does not have an instruction set) but goes really fast for less money but you are asking about a 64 bit processor with a gpu.
Everything you need to know can be easily found for free on the net. You just need to work your way through the education process of instruction sets, hdl languages, the free and the serious hdl tools, cheap and not cheap sim solutions. The areas of the world that have populations of chip engineers with some percentage on furlow as it is a rollercoaster business, company x is making a new chip, suck in 50-100 people for a few years, lay them off. Company y is making a new chip, suck in 5-100 people, then lay them of. Repeat. You can be company z if you like.