The book "Building Embedded Linux systems by Karim Yaghmour, Jon Jason Masters, Brittain Gilad and Ben-Yossef, Ian F. Darwin and Philippe Gerum" is one of early book I read when I started working with embedded linux during the year 2009. This helps me in understanding the linux system components, what is toolchain, building kernel, building rootfs content and programming startup scripts to launch the application. Along with this, I build through linux from scratch http://www.linuxfromscratch.org/ manually.
The book start with describing what is linux and goes through how to build a minimalistic embedded linux. It explains choosing between variants of components say mdev for udev, libc etc. It also explains in detail about various filesystem like jffs2, squashfs etc. And also talks about various tools to debug.
Though these days I am using Yocto for building embedded linux systems, the concepts, that I have learned with help of this book, solves many problems in my day to day activities. Though some of the tools or libraries mentioned in the book are outdated, still, it is one of the best book I would ever recommend to beginners and would recommend try to set up linux system without using tools like build-root or yocto to master embedded Linux building.