Portugal Capture Station

Task

Establish a Red Hen capture station in Portugal.

Status

Solved.
Note: instructions on this page are based on instructions provided on the page for Slick Capture.
We now have a fully automated capture system based on a tiny $40 Raspberry Pi -- a credit-card-sized motherboard:
Fenix with hard drive

For the executive version, see How to set up a Red Hen capture station.

Parsing tools

Install Debian Linux on a Raspberry Pi 2B

By José Fonseca, Polytechnic Institute of Guarda, Portugal

Raspbian installation

  1. Equipment: Mac OS X Yosemite, Raspberry Pi 2 B, HDHomeRun Dual (European Version), . . .
  2. Following the setup procedures described under —Slick capture
  3. Steps in progress now:
    1. Install Raspbian Jessie into a 32GB SanDisk Ultra SD card
      1. Download the latest version of Raspbian. Currently Jessie from https://www.raspberrypi.org/downloads/raspbian/The file is in https://downloads.raspberrypi.org/raspbian_latest
      2. Unzip the file from a Mac OS X terminal window
        3. $ unzip 2015-09-24-raspbian-jessie.zip
      3. Install The Raspbian in the SD card based on http://wiki.openelec.tv/index.php/HOW-TO:Installing_OpenELEC/Writing_The_Disk_Image#tab=Mac_OSX
        1. Insert the SD card into the Mac
        2. Find which disk number is the SD card
          3. $ diskutil list
        3. Found that the SD card is disk4. Unmount the SD card disk
          4. $ diskutil unmountDisk /dev/disk4
        4. Zero out the partition map of the SD card
          5. $ sudo dd if=/dev/zero of=/dev/rdisk4 bs=1024 count=1
        5. Write the Raspbian Wheezy disk image to the SD card
          6. $ sudo dd if=2015-09-24-raspbian-jessie.img of=/dev/rdisk4 bs=4m
        6. Ensure the changes are synced to the SD card before removing it
          7. $ sync
        7. Remove the SD card from the Mac computer
      4. Configure the Raspbian
        1. Connect the Raspberry Pi to a HDMI TV, with an Ethernet cable connected to the LAN and a USB keyboard in one UBS port
        2. Upon boot, configure the following options in the Raspberry Pi Software Configuration Tool (raspi-config) that shows automatically in the TV
          #1 Expand Filesystem
          #3 Enable Boot to Desktop/Scratch (Desktop Log in as user pi at the graphical desktop)
          #4 Internationalisation Options (to choose Portugal as the location and portuguese keyboard)
          #8 Advanced Options A2 Hostname (fenix)
          #8 Advanced Options A0 Update
        3. Reboot the Raspberry Pi
      5. Connect to the Raspberry Pi remotely, using ssh
        1. Since I changed the Hostname from the Raspberry Pi Software Configuration Tool I can access it using the Hostname using a terminal in my Mac
          2. The default login username is "pi" and the default password is "raspberry". The default password was changed immediately
          I have installed in my Mac the XQuartz X11 so I can execute graphical programs from the terminal window in the Mac
          For that I use the -X or the -Y option of the ssh
          $ ssh 193.137.162.96 -Y -l pi
          password: <secret_pass>
        1. Install vlc for the experiments
          2. $ sudo apt-get install vlc
      6. After boot, mount the external drive that was configured according to (https://sites.google.com/site/distributedlittleredhen/home/the-cognitive-core-research-topics-in-red-hen/the-barnyard/-slick-capture):

                $ sudo mount /dev/sda1 /mnt/HD1


Options for extending the life of the SD card

See discussion at Advanced Users. It may be worth moving the system logs to a USB drive to spare the SD card.
If system logs are moved to USB, we may be able to physically set the little tab on the SD card to a locked position to prevent any changes, as a security measure; see Protect your Raspberry Pi SD card with a read-only file system.

Backup and restore


Backup of the SD Card, according to (https://www.raspberrypi.org/documentation/linux/filesystem/backup.md):

sudo dd bs=4M if=/dev/mmcblk0 of=raspbian.img

Or with a better view of the process, using pipe viewer:

sudo pv -tpreb /dev/mmcblk0 -s 30908350464 |\
 dd of=/mnt/HD1/RPi_backups/backup_fenix_20151212_230304.img bs=1M conv=sync,noerror iflag=fullblock

-- where 30908350464 is the size of the SD card. This is implemented in a community-developed backup script, called sdbackup.sh on fenix, which must be run as user root:

sdbackup.sh

See also MicroSD backup and Preparing a custom image below.

Restore of the backup, according to (https://www.raspberrypi.org/documentation/linux/filesystem/backup.md):

sudo dd bs=4M if=raspbian.img of=/dev/sdb

Before we can automate the capture process, we need to configure the operating system and install the Red Hen capture scripts.

Operating System configuration

By Francis Steen
A Red Hen capture station uses a customized environment of Debian Linux.

Debian repositories

Create the following list of repositories with the command "sudo nano /etc/apt/sources.list":


# RPI Sources.list 2015-06-20 

# To sign keys -- 
# KEY=07DC563D1F41B907 
# gpg --keyserver pgp.mit.edu --recv-keys $KEY 
# gpg --keyserver subkeys.pgp.net --recv-keys $KEY 
# gpg --armor --export $KEY | apt-key add - 

# Raspbian Stable
#deb http://mirrordirector.raspbian.org/raspbian/ jessie main contrib non-free rpi
deb http://mirror.ox.ac.uk/sites/archive.raspbian.org/archive/raspbian/ jessie main contrib non-free rpi
# Uncomment line below then 'apt-get update' to enable 'apt-get source'
deb-src http://archive.raspbian.org/raspbian/ jessie main contrib non-free rpi

# Raspbian Testing
#deb http://mirrordirector.raspbian.org/raspbian/ stretch main contrib non-free rpi
deb http://mirror.ox.ac.uk/sites/archive.raspbian.org/archive/raspbian/ stretch main contrib non-free rpi
deb-src http://archive.raspbian.org/raspbian/ stretch main contrib non-free rpi

# Debian Multimedia (http://www.deb-multimedia.org/) -- unstable used for handbrake-cli 2015-11-24
deb http://www.deb-multimedia.org jessie main non-free
#deb http://www.deb-multimedia.org stretch main non-free
#deb http://www.deb-multimedia.org sid main non-free

Since we need a few packages from testing (currently stretch), let's set pinning values in /etc/apt/preferences:

# RPI Preferences 2015-11-24 
# Pinning values: see https://wiki.debian.org/AptPreferences

# apt-get install xmltv/unstable -- only the package, not the dependencies
# apt-get -t unstable install xmltv -- also the dependencies
# apt-cache policy -- inspect priorities as set here

# This sets the priorities correctly for stable (jessie), but not for testing and unstable.
# The order is critical -- FIXME!

# Unstable
Package: *
Pin: release a=unstable
Pin-Priority: -1

# Testing
Package: *
Pin: release a=testing
Pin-Priority: 10

# Deb-multimedia
Package: *
Pin: origin www.deb-multimedia.org
Pin-Priority: 100

# Debian
Package: *
Pin: origin http.debian.net
Pin-Priority: 500

# Stable
Package: *
Pin: release a=stable
Pin-Priority: 700

Package installation

  • apt-get update
  • apt-get purge wolfram-engine penguinspuzzle scratch dillo squeak-vm squeak-plugins-scratch sonic-pi idle idle3 (more)
  • apt-get purge bluej greenfoot minecraft-pi claws-mail (and their desktop files in /usr/share/raspi-ui-overrides/applications)
  • apt-get purge samba-common samba-libs vlc-plugin-samba
  • apt-get upgrade
  • apt-get install wajig alpine exim4 screen uuid dos2unix bc whois nmap fail2ban ffmpeg dvb-apps w-scan autossh
  • ln -s /usr/bin/wajig /usr/bin/just ; sudo ln -s /usr/bin/alpine /usr/bin/pine
  • just install moreutils hexedit mp4v2-utils gpac project-x x11vnc tightvncserver smartmontools lynx kpartx
  • apt-get -t stretch install xmltv (for freeguide—pulls in a bunch of perl 5.20 packages)
  • apt-get -t stretch install handbrake-cli (from marillat)
  • just reconfigure locales
  • just reconfigure exim4-config

For the upgrade to Raspberry Pi 3B, add

  • just update
  • just install piclone geany usb-modeswitch pi-bluetooth
  • just install python-pigpio python3-pigpio

Install a package from source

We cannot use packages directly from Debian, since they are compiled for ARM v7 and Raspbian is compiled for ARM v6 (Raspberry Pi 2B and 3B support both). However, in some cases, a package may be available in Debian that is not (yet) available in Raspbian. In that case, we can install the package from the Debian source package, as follows.

First make sure that the external drive is mounted so that files can be executed on it. In /etc/fstab, add "exec" after "user":

LABEL=HD1       /mnt/HD1        xfs     auto,nofail,rw,user,exec,relatime 0 0

Unmount when the drive is idle and remount. 
Second, install any dependencies the package needs.  Issue

apt-get build-dep dvblast

In the case of dvblast, also install these dependencies:

just install libbitstream-dev/stretch libev-dev/stretch

Third, activate the Debian repository in /etc/apt/sources.list, for instance

# Debian experimental

deb http://ftp.debian.org/debian experimental main contrib non-free

deb-src http://ftp.debian.org/debian experimental main contrib non-free


It may be useful to include not only deb-src but also deb, to see which packages are available with "just available", for instance "just available dvblast". When you have added the repository, comment out all the other repositories to prevent any confusion about which version to download. Run "just update" as user root.

Then switch to user csa and issue:

md ~/software/dvblast
cd ~/software/dvblast
apt-get source dvblast


This will download and uncompress the dvblast package. Walk into the uncompressed package and issue

dpkg-buildpackage -rfakeroot -us -uc

This will build a Debian package of the application. 

Switch back to user root, install the new package and copy it to any other RPi that needs it. Make sure you also revert your changes to /etc/apt/sources.list and run "just update".

Here is an example build log for dvblast, generating dvblast_3.1-1_armhf.deb:

csa@aton:~/software/dvblast/dvblast-3.1 $ dpkg-buildpackage -rfakeroot -us -uc

dpkg-buildpackage: source package dvblast

dpkg-buildpackage: source version 3.1-1

dpkg-buildpackage: source distribution experimental

dpkg-buildpackage: source changed by Sebastian Ramacher <sramacher@debian.org>

dpkg-buildpackage: host architecture armhf

 dpkg-source --before-build dvblast-3.1

 fakeroot debian/rules clean

dh clean

   dh_testdir

   dh_auto_clean

        make -j1 distclean

make[1]: Entering directory '/mnt/HD1/software/dvblast/dvblast-3.1'

CLEAN   dvblast dvblastctl dvblast.o util.o dvb.o udp.o asi.o demux.o output.o en50221.o comm.o mrtg-cnt.o asi-deltacast.o util.o dvblastctl.o

rm -f dvblast dvblastctl dvblast.o util.o dvb.o udp.o asi.o demux.o output.o en50221.o comm.o mrtg-cnt.o asi-deltacast.o util.o dvblastctl.o

make[1]: Leaving directory '/mnt/HD1/software/dvblast/dvblast-3.1'

   dh_autoreconf_clean

   dh_clean

 dpkg-source -b dvblast-3.1

dpkg-source: info: using source format `3.0 (quilt)'

dpkg-source: info: building dvblast using existing ./dvblast_3.1.orig.tar.bz2

dpkg-source: info: building dvblast in dvblast_3.1-1.debian.tar.xz

dpkg-source: info: building dvblast in dvblast_3.1-1.dsc

 debian/rules build

dh build

   dh_testdir

   dh_update_autotools_config

   dh_autoreconf

   dh_auto_configure

   dh_auto_build

        make -j1

make[1]: Entering directory '/mnt/HD1/software/dvblast/dvblast-3.1'

CC      dvblast.c

cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -Wformat-security -Wno-strict-aliasing -DVERSION=\"3.1\" -DVERSION_MAJOR=3 -DVERSION_MINOR=1 -DVERSION_EXTRA=\"release\" -D_FORTIFY_SOURCE=2 -c dvblast.c

CC      util.c

cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -Wformat-security -Wno-strict-aliasing -DVERSION=\"3.1\" -DVERSION_MAJOR=3 -DVERSION_MINOR=1 -DVERSION_EXTRA=\"release\" -D_FORTIFY_SOURCE=2 -c util.c

CC      dvb.c

cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -Wformat-security -Wno-strict-aliasing -DVERSION=\"3.1\" -DVERSION_MAJOR=3 -DVERSION_MINOR=1 -DVERSION_EXTRA=\"release\" -D_FORTIFY_SOURCE=2 -c dvb.c

CC      udp.c

cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -Wformat-security -Wno-strict-aliasing -DVERSION=\"3.1\" -DVERSION_MAJOR=3 -DVERSION_MINOR=1 -DVERSION_EXTRA=\"release\" -D_FORTIFY_SOURCE=2 -c udp.c

CC      asi.c

cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -Wformat-security -Wno-strict-aliasing -DVERSION=\"3.1\" -DVERSION_MAJOR=3 -DVERSION_MINOR=1 -DVERSION_EXTRA=\"release\" -D_FORTIFY_SOURCE=2 -c asi.c

CC      demux.c

cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -Wformat-security -Wno-strict-aliasing -DVERSION=\"3.1\" -DVERSION_MAJOR=3 -DVERSION_MINOR=1 -DVERSION_EXTRA=\"release\" -D_FORTIFY_SOURCE=2 -c demux.c

CC      output.c

cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -Wformat-security -Wno-strict-aliasing -DVERSION=\"3.1\" -DVERSION_MAJOR=3 -DVERSION_MINOR=1 -DVERSION_EXTRA=\"release\" -D_FORTIFY_SOURCE=2 -c output.c

CC      en50221.c

cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -Wformat-security -Wno-strict-aliasing -DVERSION=\"3.1\" -DVERSION_MAJOR=3 -DVERSION_MINOR=1 -DVERSION_EXTRA=\"release\" -D_FORTIFY_SOURCE=2 -c en50221.c

CC      comm.c

cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -Wformat-security -Wno-strict-aliasing -DVERSION=\"3.1\" -DVERSION_MAJOR=3 -DVERSION_MINOR=1 -DVERSION_EXTRA=\"release\" -D_FORTIFY_SOURCE=2 -c comm.c

CC      mrtg-cnt.c

cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -Wformat-security -Wno-strict-aliasing -DVERSION=\"3.1\" -DVERSION_MAJOR=3 -DVERSION_MINOR=1 -DVERSION_EXTRA=\"release\" -D_FORTIFY_SOURCE=2 -c mrtg-cnt.c

CC      asi-deltacast.c

cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -Wformat-security -Wno-strict-aliasing -DVERSION=\"3.1\" -DVERSION_MAJOR=3 -DVERSION_MINOR=1 -DVERSION_EXTRA=\"release\" -D_FORTIFY_SOURCE=2 -c asi-deltacast.c

LINK    dvblast

cc -Wl,-z,relro -o dvblast dvblast.o util.o dvb.o udp.o asi.o demux.o output.o en50221.o comm.o mrtg-cnt.o asi-deltacast.o -lpthread -lev -lrt

CC      dvblastctl.c

cc -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wall -Wformat-security -Wno-strict-aliasing -DVERSION=\"3.1\" -DVERSION_MAJOR=3 -DVERSION_MINOR=1 -DVERSION_EXTRA=\"release\" -D_FORTIFY_SOURCE=2 -c dvblastctl.c

LINK    dvblastctl

cc -Wl,-z,relro -o dvblastctl util.o dvblastctl.o -lrt

make[1]: Leaving directory '/mnt/HD1/software/dvblast/dvblast-3.1'

   dh_auto_test

 fakeroot debian/rules binary

dh binary

   dh_testroot

   dh_prep

   dh_auto_install

        make -j1 install DESTDIR=/mnt/HD1/software/dvblast/dvblast-3.1/debian/dvblast AM_UPDATE_INFO_DIR=no

make[1]: Entering directory '/mnt/HD1/software/dvblast/dvblast-3.1'

INSTALL dvblast.1 -> /mnt/HD1/software/dvblast/dvblast-3.1/debian/dvblast/usr/share/man/man1

install -m 644 dvblast.1 "/mnt/HD1/software/dvblast/dvblast-3.1/debian/dvblast/usr/share/man/man1"

INSTALL dvblast dvblastctl dvblast_mmi.sh -> /mnt/HD1/software/dvblast/dvblast-3.1/debian/dvblast/usr/bin

install dvblast dvblastctl dvblast_mmi.sh "/mnt/HD1/software/dvblast/dvblast-3.1/debian/dvblast/usr/bin"

make[1]: Leaving directory '/mnt/HD1/software/dvblast/dvblast-3.1'

   dh_installdocs

   debian/rules override_dh_installchangelogs

make[1]: Entering directory '/mnt/HD1/software/dvblast/dvblast-3.1'

dh_installchangelogs NEWS

make[1]: Leaving directory '/mnt/HD1/software/dvblast/dvblast-3.1'

   dh_installman

   dh_perl

   dh_link

   dh_strip_nondeterminism

   dh_compress

   dh_fixperms

   dh_strip

   dh_makeshlibs

   dh_shlibdeps

   dh_installdeb

   dh_gencontrol

   dh_md5sums

   dh_builddeb

dpkg-deb: building package `dvblast' in `../dvblast_3.1-1_armhf.deb'.

 dpkg-genchanges  >../dvblast_3.1-1_armhf.changes

dpkg-genchanges: including full source code in upload

 dpkg-source --after-build dvblast-3.1

dpkg-buildpackage: full upload (original source is included)


Configure locales and mta

Install the appropriate locales, including en_US.UTF-8, en_US, and pt_PT.UTF-8. Set default locale to en_US.UTF-8.

For the exim4 mail transport agent (mta), define a fully qualified domain name, e.g., fenix.universia.pt, in /etc/mailname. Also add this line to /etc/hosts (fully qualified domain name, used by alpine):

193.137.xxx.xx	fenix.universia.pt	fenix

Configure the timezone

Configure the local timezone with tzselect, if that wasn't done on installation—Red Hen capture stations should use the local timezone, not UTC. See detailed instructions. Create local user csa:csa and add this to ~/.profile:

# Time zone (set with tzselect)
TZ='Europe/Lisbon'; export TZ

Since the recording system may be unattended for long periods, also set the system time zone in the OS:

root@fenix:~# dpkg-reconfigure tzdata

Current default time zone: 'Europe/Lisbon'
Local time is now:      Tue Mar 29 16:59:29 WEST 2016.
Universal Time is now:  Tue Mar 29 15:59:29 UTC 2016.

Configure the network time protocol daemon -- queries atomic clocks on the web:

dpkg-reconfigure ntp

Verify:


csa@fenix ~ $ timedatectl
Warning: Ignoring the TZ variable. Reading the system's time zone setting only.

      Local time: Tue 2016-03-29 17:00:29 WEST
  Universal time: Tue 2016-03-29 16:00:29 UTC
        RTC time: n/a
       Time zone: Europe/Lisbon (WEST, +0100)
     NTP enabled: no
NTP synchronized: yes
 RTC in local TZ: no
      DST active: yes
 Last DST change: DST began at
                  Sun 2016-03-27 00:59:59 WET
                  Sun 2016-03-27 02:00:00 WEST
 Next DST change: DST ends (the clock jumps one hour backwards) at
                  Sun 2016-10-30 01:59:59 WEST
                  Sun 2016-10-30 01:00:00 WET

Note that Raspberry Pis do not have a hardware clock, so each time it is rebooted, it needs to read the time from the Internet.

NTP uses port 123 to communicate with time servers. Where this port is blocked, add this line to root's crontab to set the time at some interval, such as once an hour:

date -s "$(wget -qSO- --max-redirect=0 google.com 2>&1 | grep Date: | cut -d' ' -f5-8)Z"

Alternatively, use dvbdate from dvb-tools to set the time (untested). This is the preferred solution, since it's not dependent on a live Internet connection.

Finally, make sure you restart the cron daemon, so that it will use the new timezone:

service cron restart

If you forget, the crontabs of the recording schedule will use the previous timezone.

dvb-dvbt-ts

Download and compile dvb-dvbt-ts, a perl utility for repairing television transport stream files:
just install libpod-coverage-perl libtest-pod-coverage-perl
wget http://search.cpan.org/CPAN/authors/id/S/SD/SDPRICE/Linux-DVB-DVBT-TS-0.08.tar.gz
tar zxvf *gz
mv Linux-DVB-DVBT-TS-0.08 dvb-dvbt-ts-0.08
cd dvb-dvbt-ts-0.08/clib/libmpeg2
wget https://raw.githubusercontent.com/Distrotech/libmpeg2/master/libmpeg2/motion_comp_arm_s.S  (missing file)
nano Subdir.mk  -- add a line for compiling motion_comp_arm.o:

     $(libmpeg2)/motion_comp_mmx.o \
     $(libmpeg2)/motion_comp_arm.o \

cd ../..
perl Makefile.PL
dh_make -s -f ../Linux-DVB-DVBT-TS-0.08.tar.gz -e your@e-mail
debuild -i -us -uc -b

Install the resulting .deb file.

Other configurations

Create the file /etc/redhen.config with these local parameter values:

# Red Hen configuration file
location=IPG Portugal
hostname=fenix.universia.pt
ip_address=193.137.xxx.xx
timezone=Europe/Lisbon
e-mail=csa

In /etc/lightdm/lightdm.conf, set

autologin-user=csa
and add this: 
touch ~/.Xresources

Because fenix filters the VNC ports, we will need to use VNC tunnelling. If the Raspberry Pi is located behind a firewall that filters all ports, we will need to use ssh tunneling; a good solution is autossh (see instructions):

   autossh -M 20000 -f -N cartago -R 1234:localhost:22 -C

For instructions, see Securing VNC with SSH.

Install and configure fail2ban:

  • cp /etc/fail2ban/fail2ban.conf /etc/fail2ban/fail2ban.local
  • cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
  • add ignoreip = 127.0.0.1/8 164.67.171.0/24 164.67.183.179
  • service fail2ban stop
  • service fail2ban start

Turn off password login (/etc/ssh/sshd_config) to secure the system. Verify with nmap that no unnecessary ports are open; if you run VNC, shut it down between use. Unless you need an ssh tunnel, do not allow any outbound ssh connections. Change /etc/motd to a welcome message, say

       Welcome to Red Hen Lab's Raspberry Pi 2B recording station at the Instituto Politécnico da Guarda in Portugal

The HDHomerun device we are using with fenix is HDHR4-2DT, model hdhomerun4_dvbt. We've installed both the drivers provided by Silicon Dust, and I also installed the native Linux driver dvbhdhomerun. This involves building an out-of-tree kernel module, dvb_hdhomerun. I discovered we are running the 4.1.13-v7+ kernel, not sure why or how we got such a recent kernel. I couldn't find the source, and  then found rpi-update, which is used by developers to test new kernels and firmware. Before I realized this was not intended for regular users, I ran rpi-update, which may have updated the firmware and reinstalled the same 4.1.13-v7+ kernel. I still couldn't find the source package, but then found the headers. Installing it prompted for the installation of gcc-4.7, which reassuringly installed from raspbian jessie. The packages then installed and built the new module:

just install cmake libhdhomerun-dev dkms dh-systemd module-assistant
git clone https://github.com/h0tw1r3/dvbhdhomerun
cd dvbhdhomerun && dpkg-buildpackage -b
just install dvbhdhomerun-dkms_0.0.16+nmu3_all.deb 
just install dvbhdhomerun-utils_0.0.16+nmu3_armhf.deb
The dvbhomerun provides drivers for the HDHomerun devices that are fully integrated into the Linux digital television framework. This means we can use the usual Linux tools such as w_scan and gnutv -- tools we've used for years at the capture stations at UCLA. Importantly, we may also be able to use tvheadend, which provides a web-based scheduling tool that connects with the xmltv EPG. If the driver fails, we can remove the packages and maybe downgrade the kernel.

Configure the GUI

Raspbian comes with an LXDE menu; to customize, use the menu editor and see Adding LXDE start menu entries and desktop shortcuts. Menu entries are defined in .desktop file located in /usr/share/applications/usr/share/applications; they can be added and edited. For Raspberry Pi 3B (see updates), we cloned the fenix image and added the Bluetooth plugin for the lxpanel taskbar by right-clicking on the taskbar and following the menu selections. You can then click on the Bluetooth icon in the taskbar, make the RPi discoverable, scan for devices, and pair them. Once paired, the devices should automatically reattach on reboot.

If the devices do not reattach on reboot, or if you want access to the GUI from another computer for any reason, you can run VNC on the RPi by issuing

   vncserver -geometry 800x600 -depth 24 :1

On your laptop, to create a safe connection, forward the VNC port for screen 1:

  ssh fenix.local -nNT -L 5991:localhost:5901 -l csa

You can then connect to the Raspberry Pi desktop with this command on a Mac:

   open vnc://fenix.local:5901

or run any VNC client.

Set up the HDHomeRun

By Jose Fonseca, Polytechnic Institute of Guarda, Portugal
  1. Connect the HDHomeRun to my LAN Switch where I also have the Raspberry Pi connected
      2. In my network the HDHomeRun has the hostname: HDHR-123008C7
  2. Connect the HDHomeRun to the cable TV
    1. Install and configure HDHomeRun, based on —Slick capture
      1. Install the HDHomeRun into the Raspberry Pi
          2. The software is in this page http://www.silicondust.com/support/downloads/linux/
          From a terminal window connected to the Raspberry Pi download the file http://download.silicondust.com/hdhomerun/libhdhomerun_20150615.tgz
          $ wget http://download.silicondust.com/hdhomerun/libhdhomerun_20150615.tgz
          Extract it
          $ tar -xvzf libhdhomerun_20150615.tgz
          From a terminal window connected to the Raspberry Pi download the file http://download.silicondust.com/hdhomerun/hdhomerun_config_gui_20150615.tgz
          $ wget http://download.silicondust.com/hdhomerun/hdhomerun_config_gui_20150615.tgz
          extract it
          $ tar -xvzf hdhomerun_config_gui_20150615.tgz
          Install the libgtk library needed to install the HDHomeRun $ sudo apt-get install libgtk2.0-dev
          cd hdhomerun_config_gui
          sudo ./configure
          sudo make
          sudo make install
          cd ..
      2. Configure the HDHomeRun
          3. Search for the HDHomeRun on the subnet
          $ hdhomerun_config discover
          Response: hdhomerun device 123008C7 found at 192.168.1.31
          Find the model of your HDHomeRun
          $ hdhomerun_config 123008C7 get /sys/model
          Response: hdhomerun4_dvbt
          Find the firmware version of your HDHomeRun
          $ hdhomerun_config 123008C7 get /sys/version
          Response: 20150505
          Set a variable for DEVICE and a variable for TUNER
          $ DEVICE=123008C7 TUNER=0
          Get the channelmap
          $ hdhomerun_config $DEVICE get /tuner0/channelmap
          Response: eu-bcast
        1. List channels
          2. Scan tuner 0 for channels
          $ hdhomerun_config $DEVICE scan /tuner$TUNER scan0eu-cable.log
          Response:

          SCANNING: 858000000 (eu-bcast:69)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 850000000 (eu-bcast:68)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 842000000 (eu-bcast:67)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 834000000 (eu-bcast:66)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 826000000 (eu-bcast:65)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 818000000 (eu-bcast:64)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 810000000 (eu-bcast:63)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 802000000 (eu-bcast:62)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 794000000 (eu-bcast:61)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 786000000 (eu-bcast:60)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 778000000 (eu-bcast:59)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 770000000 (eu-bcast:58)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 762000000 (eu-bcast:57)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 754000000 (eu-bcast:56)

          LOCK: t8qam64 (ss=79 snq=77 seq=100)

          TSID: 0x044D

          PROGRAM 1101: 1 RTP 1

          PROGRAM 1102: 2 RTP 2

          PROGRAM 1103: 3 SIC

          PROGRAM 1104: 4 TVI

          PROGRAM 1107: 5 ARTV

          PROGRAM 1111: 8 HD

          SCANNING: 746000000 (eu-bcast:55)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 738000000 (eu-bcast:54)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 730000000 (eu-bcast:53)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 722000000 (eu-bcast:52)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 714000000 (eu-bcast:51)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 706000000 (eu-bcast:50)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 698000000 (eu-bcast:49)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 690000000 (eu-bcast:48)

          LOCK: t8qam64 (ss=66 snq=62 seq=100)

          TSID: 0x044D

          PROGRAM 1101: 1 RTP 1

          PROGRAM 1102: 2 RTP 2

          PROGRAM 1103: 3 SIC

          PROGRAM 1104: 4 TVI

          PROGRAM 1107: 5 ARTV

          PROGRAM 1111: 8 HD

          SCANNING: 682000000 (eu-bcast:47)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 674000000 (eu-bcast:46)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 666000000 (eu-bcast:45)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 658000000 (eu-bcast:44)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 650000000 (eu-bcast:43)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 642000000 (eu-bcast:42)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 634000000 (eu-bcast:41)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 626000000 (eu-bcast:40)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 618000000 (eu-bcast:39)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 610000000 (eu-bcast:38)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 602000000 (eu-bcast:37)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 594000000 (eu-bcast:36)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 586000000 (eu-bcast:35)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 578000000 (eu-bcast:34)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 570000000 (eu-bcast:33)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 562000000 (eu-bcast:32)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 554000000 (eu-bcast:31)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 546000000 (eu-bcast:30)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 538000000 (eu-bcast:29)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 530000000 (eu-bcast:28)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 522000000 (eu-bcast:27)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 514000000 (eu-bcast:26)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 506000000 (eu-bcast:25)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 498000000 (eu-bcast:24)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 490000000 (eu-bcast:23)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 482000000 (eu-bcast:22)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 474000000 (eu-bcast:21)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 226500000 (eu-bcast:12)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 219500000 (eu-bcast:11)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 212500000 (eu-bcast:10)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 205500000 (eu-bcast:9)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 198500000 (eu-bcast:8)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 191500000 (eu-bcast:7)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 184500000 (eu-bcast:6)

          LOCK: none (ss=0 snq=0 seq=0)

          SCANNING: 177500000 (eu-bcast:5)

          LOCK: none (ss=0 snq=0 seq=0)


        2. Capture Portuguese channels
          3. $ hdhomerun_config $DEVICE set /tuner$TUNER/channel auto:56
        3. View the status:
          4. $ hdhomerun_config $DEVICE get /tuner$TUNER/status
        4. Response:

          5. ch=auto:56 lock=t8qam64 ss=80 snq=72 seq=100 bps=19854304 pps=0


          From the channels list we can see that the list of portuguese channels is:
          • PROGRAM 1101: 1 RTP 1
          • PROGRAM 1102: 2 RTP 2
          • PROGRAM 1103: 3 SIC
          • PROGRAM 1104: 4 TVI
          To capture, for example RTP1, we need to set the tuner to program 1101:
          $ hdhomerun_config $DEVICE set /tuner$TUNER/program 1101
          Then start the capture:
          $ hdhomerun_config $DEVICE save /tuner$TUNER TestCaptureRTP1.ts
          Response:
          (stop the recording with control-C)
          ...................................................................................................................................^C
          -- Video statistics --
          27791 packets received, 0 overflow errors, 0 network errors, 0 transport errors, 0 sequence errors
          The file generated may be transferred to a remote linux based computer by using scp from a terminal in that computer:
          $ scp  pi@193.137.162.96:"/home/pi/TestCaptureRTP1.ts" "/remote_computer_folder_path/" 

      TV capture configuration

      by Francis Steen

      I've added the Red Hen scripts and directories, as follows:

      • chown -R csa:csa /mnt/HD1
      • mkdir /mnt/HD1/tv ; ln -s /mnt/HD1/tv /tv  # Root of the repository tree
      • mkdir -p /mnt/HD1/tvspare/tuners ; ln -s /mnt/HD1/tvspare /tvspare  # Used for tuner allocation and log
      • mkdir /mnt/HD1/ts ; ln -s /mnt/HD1/ts /home/csa/ts  # Used for transport stream file repair

      I built ccextractor-0.78 and copied these scripts to /local/user/bin:

      • -rwxr-xr-x 1 csa  csa    5828 Dec 25 08:58 cc-extract-bin.sh
      • lrwxrwxrwx 1 root csa      25 Dec  6 21:30 ccextractor -> /usr/bin/ccextractor-0.78
      • -rwxr-xr-x 1 csa  csa   12270 Dec 25 13:30 cc-extract-teletext.sh
      • -rwxr-xr-x 1 csa  csa    1333 Dec  4 22:54 channel
      • -rwxr-xr-x 1 csa  csa    7643 Dec 25 13:43 channel_hdhr_2015-12-05.sh
      • -rwxr-xr-x 1 csa  csa      27 Dec  5 19:16 channel_hdhr.name
      • -rwxr-xr-x 1 csa  csa    9769 Dec  9 19:22 check-cc-single.sh
      • -rwxr-xr-x 1 csa  csa    4446 Dec  9 11:37 check-mpg-single.sh
      • lrwxrwxrwx 1 root staff    19 Nov 25 14:01 compress -> ts2mp4-single-01.sh
      • -rwxr-xr-x 1 csa  csa    1687 Dec  5 20:14 find-tuners.sh
      • lrwxrwxrwx 1 csa  csa      15 Dec  7 08:36 schedule -> scheduler-03.sh
      • -rwxr-xr-x 1 csa  csa    4585 Dec 18 12:49 scheduler-03.sh
      • -rwxr-xr-x 1 csa  csa    6075 Dec 12 23:01 sdbackup.sh
      • -rwxr-xr-x 1 csa  csa    3490 Dec  9 08:55 select-tuner-native-locking.sh
      • -rwxr-xr-x 1 csa  csa   14972 Dec  9 11:44 ts2mp4-single-01.sh
      • -rwxr-xr-x 1 csa  csa    1554 Dec  6 20:06 tunnel
      • -rwxr-xr-x 1 csa  csa    3626 Dec  5 00:34 tunneld
      • lrwxrwxrwx 1 root staff    20 Nov 29 08:15 xmltv-download -> xmltv-download-01.sh
      • -rwxr-xr-x 1 csa  csa    3790 Dec  9 11:35 xmltv-download-01.sh
      • -rwxr-xr-x 1 csa  csa     466 Dec  5 06:05 zinfo
      I then ran "find-tuners.sh", which generated /tvspare/tuners/signal, which I copied to /tvspare/tuners/current. This list of tuners is used to assign the two tuners dynamically to each new recording process. The system is designed to handle a pool of HDHR3s shared between several recording machines and is overkill for the RPi, but should work fine—and will scale as needed.

      The recording system uses crontabs for scheduling. The syntax is as follows (see channel -h):

           channel 1, 30min, "Program name", 1

      where channel is the name of the script, 1 is the RTP-1 program 1101, "Program name" you fill in, and 1 is just a marker for our online schedule validator, which we may or may not end up activating.

      Since /usr/local/bin/channel just links to the real script, you can do this to see the details:

           bash -xv channel_hdhr_2015-11-06.sh 1, 2min, TEST, 1

      Files are recorded to /mnt/spool. Ideally, this would be an external harddrive, to remove some of the load on the solid-state drive. It could even be a 7200rpm externally powered drive that would be fast enough to handle two simultaneous recordings, though this is not of first importance.

      A crontab entry might look like this:

      #JUNE 2015

      # CSPAN1 061615 TUE  Donald Trump Presidential Campaign Announcement

      1 3 17 6 *       channel 58, 60min, "US Presidential Politics", 3, "Donald Trump Presidential Campaign Announcement"

      I created a crontab for user csa with a single-run test example. 

      When it ran, it also generated an e-mail receipt:

      Date: Fri, 06 Nov 2015 20:19:57 +0000

      From: Cron Daemon <root@fenix>

      To: csa@fenix

      Subject: Cron <csa@fenix> channel 1, 10min, "TEST", 1, "Optional comment"

      Fenix initiating capture on Sexta 06 Novembro 2015 at 20:10:01 with tuner 123008C7-0, duration 595 seconds

              2015-11-06_2010_PT_RTP-1_TEST

      Fenix completed recording on Sexta 06 Novembro 2015 at 20:19:57

      -rw-r--r-- 1 csa csa 203M Nov  6 20:19 /mnt/spool/2015-11-06_2010_PT_RTP-1_TEST.mpg

      -rw-r--r-- 1 csa csa   49 Nov  6 20:19 /tv/2015/2015-11/2015-11-06/2015-11-06_2010_PT_RTP-1_TEST.t

      You can see these e-mails in the alpine mail reader. So far, everything seems to be working, so if you could schedule some actual news programs, that would be great. 

      The file that ends in .t is a header file, used to construct the eventual teletext file, but I've not set that up yet.

      You can move into the directory tree /tv that contains the recorded files by typing

           day 

      for today, or

          day 1

      for yesterday, and so on going backwards in time. The directories are created automatically.

      Television schedules

      The list of Portuguese channels is:

      PROGRAM 1101: 1 RTP 1
      PROGRAM 1102: 2 RTP 2
      PROGRAM 1103: 3 SIC
      PROGRAM 1104: 4 TVI

      The programs are all on eu-bcast:56. Add this information to the parameter file /tvspare/tuners/lineup, along with the network ID (from the xmltv downloads in /tvspare/xmltv), the country, the language, and the most-used teletext page number:


      # Television networks at this location
      CHANNEL=1 NWK=RTP-1 CH=56 PN=1101 ID=5.nos.pt COUNTRY=PT LAN=POR TP=885
      CHANNEL=2 NWK=RTP-2 CH=56 PN=1102 ID=3.nos.pt COUNTRY=PT LAN=POR TP=884
      CHANNEL=3 NWK=SIC   CH=56 PN=1103 ID=7.nos.pt COUNTRY=PT LAN=POR TP=888
      CHANNEL=4 NWK=TVI   CH=56 PN=1104 ID=8.nos.pt COUNTRY=PT LAN=POR TP=888


      The lineup file is used as a lookup table by the various scripts that need this information.

      The three main channels (RTP, public and SIC and TVI, private channels) broadcast their evening news at 8pm. Viewers shares are highest for SIC, then TVI and the least seen is RTP (http://binaries.cdn.impresa.pt/dealer/2246924/AUDIENCIAS-Abril+20144205681231226447556.pdf)

      Here's the link to the schedules:

      SIC - http://sicnoticias.sapo.pt/programas/jornaldanoite/
      TVI - http://www.tvi.iol.pt/programa/jornal-das-8/53c6b3903004dc006243d0cf
      RTP - http://www.rtp.pt/programa/tv/p31394

      There is a show that might be of interest. Every week they choose a news topic and invite politicians, experts and other relevant players to discuss it. Prós & Contras: http://www.rtp.pt/play/p1772/Pros-e-Contras

      According to Wikipedia's list of networks with teletext, these three networks all have teletext numbers:

      Sociedade Independente de Comunicação: teletext page 888 for live captions—online teletext
      Televisão Independente: unknown teletext for live captions—online teletext
      Rádio e Televisão de Portugal: unknown teletext for live captions—online teletext

      Automating the capture schedule

      After completing the operating system and tv capture configurations, the recording schedule can be automated through crontab, as in this example for user pi:

      00 20 * * *     channel 3, 60min, "Jornal da Noite", 1, "National evening news"

      The full crontab looks like this:

      #### CRONTAB FOR CSA ON FENIX ########################################
      #
      #### PATH AND MAIL INFORMATION #######################################
      #
      # Folders to search for program files
      PATH=/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
      # Email output -- use a comma between addresses
      # Copies should be sent to the local pi user and possibly to tna@commstds.ucla.edu
      MAILTO=csa
      #
      #### CHANNELS ########################################################

      # Defined in the channel script (currently channel_hdhr_2015-11-06)

      #  1 ) NWK=RTP-1 CH=56 PN=1101 ;;
      #  2 ) NWK=RTP-2 CH=56 PN=1102 ;;
      #  3 ) NWK=SIC   CH=56 PN=1103 ;;
      #  4 ) NWK=TVI   CH=56 PN=1104 ;;

      #### AD HOC SCHEDULE #################################################

      # November 2015

      # RTP-1 Test recording
      10 20 06 11 *       channel 1, 10min, "TEST", 1, "Optional comment"
      19 21 06 11 *       channel 2, 10min, "TEST", 2, "Optional comment"
      08 00 07 11 *       channel 3, 10min, "TEST", 2, "Optional comment"

      #### REGULAR SCHEDULE #################################################

      # 20:00 SIC Main national evening news
      00 20 * * *     channel 3, 60min, "Jornal da Noite", 1, "National evening news"

      #### CRONTAB SYNTAX ###############################################
      #
      # ------------- minute (00 - 59)
      # | ----------- hour (00 - 23)# | | --------- day of month (01 - 31)  
      # | | | ------- month (01 - 12)
      # | | | | ----- day of week (1 - 7) (Sunday=7)
      # | | | |
      # * * * * * command to be executed
      # 
      ###################################################################

      The script xmltv-download.sh daily fetches the broadcast schedule for the networks we're interested in. In honor of the new recording location, we wrote a new script, scheduler.sh, which examines the downloaded schedule for shows we record and automatically generates the crontab recording schedule, see Automatic scheduling.

      For the entire process to be automated, we also need the scripts that extract the text, compress the video, and copy the completed files to NewsScape, as described in the sections below.

      Text extraction

      The Portuguese television transport stream typically contains a timestamped transcript of the news show. We use CCExtractor to extract these subtitles. For how to retrieve and compile CCExtractor, see Brazil Capture Station. 

      The list of the teletext page of the transcription of the main news programs is the following (see RTP teletext):
        • RTP1 - 885
        • RTP2 - 884
        • SIC - some shows use 888
        • TVI - some shows use 888
      Note that different types of programs use different teletext pages -- in the case of RTP:
      • Página 887 - RTP1 programs
      • Página 888 - RTP2 programs
      • Página 889 - RTP Internacional programs
      • Página 886 - RTP1 captions of live football games
      • Página 884 - RTP2 automatic captions of news shows
      • Página 885 - RTP1 automatic captions of news shows
      To look for the embedded teletext, try these commands:

      FIL=2015-11-07_0008_PT_SIC_TEST
      ccextractor -out=txt $FIL.mpg -o $FIL.txt

      You may get teletext, or no text but a block of suggested teletext pages to check:

      Notice: Teletext page with possible subtitles detected: 888
      Notice: Teletext page with possible subtitles detected: 886
      Notice: Teletext page with possible subtitles detected: 883
      Notice: Teletext page with possible subtitles detected: 882
      Notice: Teletext page with possible subtitles detected: 881
      Notice: Teletext page with possible subtitles detected: 880
      Notice: Teletext page with possible subtitles detected: 867
      Notice: Teletext page with possible subtitles detected: 866

      To check each of these, issue this command on one line:

      for TP in 888 886 883 882 881 880 867 866 ; 
          do echo -e "\n\tTrying teletext page $TP:\n" ;
          ccextractor -tpage $TP -out=txt $FIL.mpg -o $FIL.txt ; sleep 1
      done

      Once you do find text, add the teletext page you identified to the cc-extract-teletext.sh script and run it:

      cc-extract-teletext 2015-11-07_0008_PT_SIC_TEST.mpg

      Our first results for SIC teletext page 888 are excellent, at least for the telenovella A Regra do Jogo:

      20151107000800.480|20151107000806.040|888|:)  - Está aqui a tua parte. - Nem sei como te agradecer.
      20151107000806.240|20151107000809.880|888|É fruto do teu trabalho. Mereceste-o.
      20151107000810.360|20151107000815.720|888|:)  - Sabes a Dona Zezela ? - Sei.
      20151107000816.160|20151107000821.520|888|- Gostou do ajuste no vestido. - Eu alonguei o vestido.
      20151107000821.920|20151107000823.800|888|Vai indicar-te às amigas.
      20151107000824.720|20151107000828.920|888|O dinheiro vem numa hora tão boa. O Juca precisa...
      20151107000829.320|20151107000833.120|888|:-/  Esse dinheiro é para ti. Para ficares bonita.
      20151107000835.920|20151107000840.080|888|:)  Tenho um vestido que te ficará muito bem.
      20151107000842.200|20151107000847.480|888|:)  - Está aqui. Olha. - Não é muito ousado ?
      20151107000847.760|20151107000852.400|888|(Indira) Experimenta-o. Dá-te cor e vida.
      20151107000854.840|20151107000856.160|888|Quero ver.
      20151107000858.080|20151107000900.480|888|:)  Eu sabia que ficaria maravilhoso.
      20151107000901.120|20151107000905.600|888|:)  - Só tenho de tirar a etiqueta. :)  - Olha os meus ombros.
      20151107000905.840|20151107000910.560|888|:)  Estás linda ! Vai para casa. Não aceito devoluções.
      20151107000910.800|20151107000914.320|888|:)  - É uma prenda ? (Indira)  - Vai para casa.
      20151107000914.520|20151107000919.800|888|:)  Veste-te bem, arranja um jeitoso que te mereça e valorize.
      20151107000923.440|20151107000927.840|888|:)  - Indira, obrigada. :)  - De nada.
      20151107000928.640|20151107000933.240|888|:)  - Dá uma volta. (Domingas)  :)  - Que vergonha.
      20151107000934.360|20151107000938.320|888|(# música ritmada - funk Dream Team do Passinho - De Ladin)
      20151107000938.560|20151107000947.440|888|(#)
      20151107000948.000|20151107000951.280|888|:-O  Tina ! Acabou-se a comida !
      20151107000951.760|20151107000957.880|888|(#)
      20151107000958.080|20151107001001.160|888|:)  - Comi tudo. (Rui)  :-s  - Tina...
      20151107001001.360|20151107001006.680|888|:)  Esquece isso. Vem cá. Olha para a luz.
      20151107001007.160|20151107001011.440|888|:-&  Que luz ? Há roupa espalhada e louça para lavar.
      20151107001011.840|20151107001016.520|888|- Lava tu a loiça. :-s  - Eu ?
      20151107001016.840|20151107001019.520|888|- Desenvencilha-te ! :-s  - Tina, tu...
      20151107001021.600|20151107001024.680|888|:-s  - O que está a acontecer ? - Decidi ouvir-te.
      20151107001024.960|20151107001028.640|888|:-s  - A mim ? - Não disseste para sermos livres ?
      20151107001030.320|20151107001035.680|888|:-&  Já não precisas de ir trabalhar para o escritório.
      20151107001035.880|20151107001040.920|888|:-&  A Tina dona de casa acabou. A Tina morreu.
      20151107001042.840|20151107001045.520|888|Agora sou a Tina do morro da macaca.
      20151107001046.240|20151107001048.920|888|(Tina canta) É a Tina Furacão
      20151107001049.160|20151107001052.880|888|:-&  - Não estás nada bem. - Estou maravilhosa.
      20151107001054.200|20151107001057.920|888|:-s  - Estás a beber ? - Desde o meio-dia.
      20151107001059.960|20151107001105.120|888|:)  A vida de favelada é demais. Comprei torresmos.
      20151107001106.880|20151107001109.440|888|:)  - Rapaz ! :-&  - Estás bêbada !
      20151107001110.000|20151107001112.280|888|(Tina) :-&  E tu estás chato !
      20151107001113.560|20151107001116.880|888|:-&  - Acabou a festa. - Está só a começar.
      20151107001117.160|20151107001121.640|888|:-&  - Onde vais ? - Vou sair à noite !
      20151107001122.080|20151107001127.840|888|:-&  Não tens a menor condição... Tina ! Tina !
      20151107001133.400|20151107001137.440|888|(Domingas)  :-&  - Olá. - Tens um vestido novo ?
      20151107001137.640|20151107001139.960|888|:-&  - Gostaste ? - É parecido contigo.
      20151107001142.120|20151107001145.640|888|Horroroso, foleiro, ridículo.
      20151107001146.600|20151107001148.160|888|:-&  Vai para o inferno.
      20151107001150.520|20151107001154.600|888|:-/  - Como ? Queres apanhar ? :-&  - Larga-me !
      20151107001154.840|20151107001156.840|888|:-/  - Dá-me o dinheiro ! :-&  - Não tenho dinheiro.
      20151107001157.040|20151107001200.320|888|:-/  - Como compraste o vestido ? :-&  - A Indira deu-mo.
      20151107001200.520|20151107001203.720|888|:-/  - Porquê ? :-&  - Porque tem pena de mim.
      20151107001203.960|20151107001206.760|888|:)  - Pena de ti ? :-&  - Ela vê como me tratas.
      20151107001206.960|20151107001212.480|888|:-&  - Um dia denunciam-te. :-/  - Antes parto-te os dentes.
      20151107001214.520|20151107001220.320|888|:-/  És uma ingrata. Devias agradecer por não te ter deixado.
      20151107001220.960|20151107001224.600|888|:-/  Há tantas mulheres jeitosas e fico contigo.
      20151107001225.160|20151107001228.760|888|(Juca) :-/  Esse cinto não combina com essa roupa.
      20151107001228.960|20151107001230.520|888|:-/  O dinheiro está aqui ?
      20151107001230.760|20151107001234.440|888|(Domingas) :(  Para ! Não tenho dinheiro.
      20151107001234.640|20151107001236.920|888|:(  - Por favor... :-/  - O dinheiro ?
      20151107001237.280|20151107001240.680|888|:(  O dinheiro é para pagar as contas da casa.
      20151107001241.600|20151107001244.960|888|:-/  Sou o homem da casa. O dinheiro é meu.
      20151107001245.080|20151107001250.240|888|:(  - Então devias ganhar dinheiro. :-/  - O que disseste ?
      20151107001250.520|20151107001252.640|888|:-/  Repete !
      20151107001257.400|20151107001300.440|888|:-/  Da próxima vez, mando-te para o hospital !
      20151107001301.000|20151107001305.280|888|(Juca) :-/  Quando eu voltar, quero tudo arrumado !
      20151107001306.800|20151107001310.920|888|:(  Dá-me o meu dinheiro. Por favor, Juca.
      20151107001311.440|20151107001315.480|888|(# música instrumental - triste) (Domingas chora)
      20151107001315.640|20151107001328.880|888|(#) (---)
      20151107001329.560|20151107001333.560|888|(# música ritmada - funk)
      20151107001333.760|20151107001346.800|888|(Merlô canta) O que é isto ? Esta mulher é muito quente
      20151107001347.040|20151107001351.440|888|(---) Ela roda e enrosca-se E faz de mim um varão de dança
      20151107001351.640|20151107001354.840|888|(---) O que é isto ? Esta mulher é muito quente
      20151107001355.040|20151107001359.000|888|(---) Ela roda e enrosca-se E faz de mim um varão de dança
      20151107001359.240|20151107001402.040|888|(---) Ela é toda boa Gosta de provocar
      20151107001402.280|20151107001405.600|888|(---) Adora fantasia Ela quer amarrar-me
      20151107001405.800|20151107001409.440|888|(---) Não vai ao ginásio Diz que sou o seu treinador
      20151107001409.680|20151107001412.480|888|(---) Que saúde Já me sinto mal
      20151107001412.720|20151107001416.000|888|(---) Já te agachaste ? E já saltaste ?
      20151107001416.200|20151107001417.640|888|(---) O que falta ?
      20151107001417.760|20151107001421.560|888|(Alisson e Ninfa cantam) Espera que eu vou mostrar
      20151107001421.720|20151107001424.400|888|(---) Fica quieto Não te movimentes
      20151107001424.640|20151107001426.720|888|(---) Vou dançar no varão
      20151107001426.920|20151107001430.120|888|(---) O que é isto ? Esta mulher é muito quente
      20151107001430.360|20151107001434.640|888|(---) Ela roda e enrosca-se E faz de mim um varão de dança
      20151107001436.320|20151107001440.120|888|(Rui) :-s  Tina ! Tina, estás maluca ?
      20151107001440.520|20151107001444.680|888|(Rui)  :-&  - Vamos embora ! (Tina)  :)  - Estou a adorar !
      20151107001444.960|20151107001448.480|888|:-&  - Que vergonha. :-/  - Deixa-me em paz !
      20151107001448.880|20151107001452.040|888|:-&  Estás louca ? Estás a expor-te !
      20151107001452.640|20151107001457.400|888|:-&  Quero que falem de mim na favela !
      20151107001457.800|20151107001501.160|888|:-&  - Vamos para casa ! :-&  - Aquela barraca ?
      20151107001502.200|20151107001505.720|888|:-&  - Vamos embora ! :-&  - Vem para o varão !
      20151107001506.400|20151107001510.360|888|:-&  - Chega ! - A barraca está uma confusão !
      20151107001510.600|20151107001515.560|888|:-&  Cheira mal e está nojenta ! Mas não há problema !
      20151107001515.760|20151107001518.280|888|:-&  Vamos ser felizes !
      20151107001518.600|20151107001522.560|888|(# música ritmada - funk)
      20151107001523.080|20151107001527.040|888|(Rui) :-s  Tina ! Tina !
      20151107001527.960|20151107001530.960|888|:)  Aqui acontece sempre o inesperado !
      20151107001531.200|20151107001532.840|888|:)  É o melhor da cidade.
      20151107001533.120|20151107001536.360|888|:)  Temos camarão e champanhe do bom.
      20151107001536.760|20151107001538.720|888|:)  - És o Juca ? :)  - Sim.
      20151107001538.960|20151107001542.000|888|:)  - O que fazes ? :)  - Sou empresário.
      20151107001542.200|20151107001547.760|888|:)  Todos me conhecem. Nunca vi o teu sorriso na macaca.
      20151107001548.360|20151107001550.920|888|:)  - Sou de Madureira. - Que beleza.
      20151107001553.160|20151107001557.800|888|:)  Estou a pensar abrir um negócio em Madureira.
      20151107001558.840|20151107001604.040|888|(Domingas) :-&  É nesta mulher que gastas o meu dinheiro ?
      20151107001604.440|20151107001608.400|888|(Juca) :-/  Estás louca ? Vai para casa !
      20151107001609.000|20151107001614.760|888|(---) :-O  Larguem-me ! Não me toquem !
      20151107001617.920|20151107001621.760|888|(# música alegre - samba)
      20151107001621.960|20151107001629.560|888|(Zé Maria) Aqueci o molho.
      20151107001630.600|20151107001634.360|888|- O molho fica à parte. :)  - Nossa Senhora.
      20151107001634.560|20151107001638.440|888|Tem muitas ervas e vinho tinto.
      20151107001639.200|20151107001643.080|888|:)  - Trouxe pão fresco. (Djanira)  :)  - Surpresa !
      20151107001643.320|20151107001647.320|888|(Djanira)  :)  - Vê quem veio ! :)  - Tudo bem, Tóia ?
      20151107001649.680|20151107001653.320|888|(Djanira) :)  Ainda bem que trouxeste o pão fresco.
      20151107001653.560|20151107001656.560|888|Tínhamos de fazer a sanduíche com pão velho.
      20151107001657.600|20151107001701.880|888|:)  Que beleza. Vai ficar muito bom.
      20151107001702.120|20151107001704.400|888|Está lindo. Desculpa.
      20151107001706.240|20151107001711.720|888|Antes do pernil que eu trouxe, quero dar-te esta prenda.
      20151107001712.280|20151107001715.680|888|- É de coração, para ti. :-s  - Para mim ?
      20151107001716.560|20151107001720.200|888|:)  Queria ter-te dado da outra vez que estive aqui.
      20151107001720.400|20151107001722.560|888|(Zé Maria) :)  Vê se gostas.
      20151107001724.400|20151107001727.520|888|:)  - É bonito. :)  - Põe ao pescoço.
      20151107001730.120|20151107001733.680|888|:)  - Obrigada. (Zé Maria)  :)  - De nada.
      20151107001734.800|20151107001737.880|888|Só penso que arrisca a vida ao vir aqui.
      20151107001738.160|20151107001741.360|888|- Porquê ? - Está a ser perseguido.
      20151107001741.800|20151107001744.760|888|A minha família vale o risco.
      20151107001746.880|20151107001752.080|888|Da última vez que estive aqui, fui-me embora chateado.
       
      The RTP-1 broadcasts Telejornal, its main evening news show, at 20:00, with excellent teletext captions. We now record this show on a regular schedule, tracked by the new schedule script.

      Video compression

      We currently transfer the uncompressed files to the Hoffman2 high-performance computing cluster at UCLA, which processes the files and sends them to the NewsScape search engines and archival servers. However, in some cases it may be necessary or desirable to perform the compression locally, either with software codecs or with the hardware codec built into the Raspberry Pi.

      Video compression is done with HandBrake or ffmpeg. Get HandBrake with NEON support:

      sudo apt-get update
      sudo apt-get upgrade
      sudo apt-get install gdebi-core
      wget http://steinerdatenbank.de/software/ghb_0.10.5-1_armhf.deb
      sudo gdebi ghb_0.10.5-1_armhf.deb

      It's supposed to encode SD video faster than real time.

      The current script uses ffmpeg, with HandBrake as a fallback if ffmpeg fails. The process is scripted:

      bash -xv ts2mp4-single-01.sh 2015-11-25_0000_PT_SIC_A_Regra_do_Jogo.mpg ; date

      On a Raspberry Pi 3B, such as odin, software compression takes around twice the length of the recording. This is fast enough to be a realistic option. On a Raspberry Pi 2B, such as redhen3rpi, software compression takes 4 hours and 39 minutes to processes a one-hour 981MB mpeg file; this is typically too slow to be useful.

      The Raspberry Pi also has a hardware compression chip. To support it, build a custom version of ffmpeg:

      ./configure --enable-gpl --enable-libx264 --enable-nonfree --enable-mmal --enable-omx --enable-omx-rpi --enable-libfdk-aac
      make -j4
      sudo make install
      ldconfig

      You can then use the ts2mp4-single-02.sh script as above and encode a one-hour video in about 20 minutes. The quality is decent, but not as good as software-encoded files. 

      Search engine access

      The files are now available to the UCLA campus and the Red Hen community of students and researchers. In the NewsScape search engine, the timestamped teletext is used as a search index and a navigational framework for the news videos:

      Portuguese news in NewsScape&#39;s Edge search engine

      Machine translation within the search engine:

      Machine translation from Portuguese to English
      Preparing a custom image
      After creating a backup image of fenix -- see Backup and restore above and MicroSD backup -- you can mount the image to inspect and modify it. Since the full image contains two active partitions, you need to mount one partition at a time. To see the partitions:

      root@fenix:/mnt/HD1/RPi_backups# kpartx -l 2015-12-25_Fenix-generalized.img
      loop0p1 : 0 114688 /dev/loop0 8192
      loop0p2 : 0 60244992 /dev/loop0 122880

      You can now create custom mount points and mount the main partition like this:

      kpartx -a 2015-12-25_Fenix-generalized.img
      mount /dev/mapper/loop0p2 /mnt/loop/ -o loop,rw

      This makes each partition of the backup image available for editing. You can mount the image partition on any Linux machine and edit the text files -- for instance, change the host name, or create a new /etc/redhen.config file. 
      You can also create a virtual fenix by changing root to the image:

      chroot /mnt/loop /bin/bash

      Now you can even run programs within the image, such as generating new rsa keys. Note that you can only do this on a Raspberry Pi -- on fenix itself or another RPi -- because the binary executables have been compiled for the ARM CPU. These commands allow us to take a full backup of fenix, and then customize the image for a new user. Some values you may want to customize:
      • /etc/hosts
      • /etc/hostname
      • /etc/mailname
      • dpkg-reconfigure tzdata
      • /etc/redhen.config
      • /tvspare/tuners/lineup
      • crontab (user csa)
      Information not known beforehand can be added once the new unit is on site.

      Change the passwords for all users as user root, using 
      • passwd $USER
      To create new RSA keys, first recreate some needed devices as root:
      • mknod /dev/null c 1 3
      • chmod 666 /dev/null
      • mknod /dev/random c 1 9
      • mknod /dev/urandom c 1 9
      • rm .ssh/id* ; ssh-keygen -t rsa
      • su csa (and any other user)
        • rm .ssh/id* ; ssh-keygen -t rsa
      This makes the distribution of images safer.

      Comments