{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fa54ff82",
   "metadata": {},
   "source": [
    "# Lecture 11 — Hands-on: the Corbet diagram and continuum lags in Mrk 110\n",
    "\n",
    "| part | topic | data |\n",
    "|---|---|---|\n",
    "| A | Correlations and two-sample tests: $P_{\\rm spin}$ vs $P_{\\rm orb}$ of HMXBs in the SMC and the Galaxy | `HMXB/*.csv` |\n",
    "| B | Time lags: ICCF, DCF, FR/RSS, detrending and the lag spectrum of Mrk 110 | `Mrk110_Swift/`, `Mrk110_ground/` |\n",
    "\n",
    "Unzip `Lecture11_class_data.zip` next to this notebook (or set `DATA` below).\n",
    "Cells marked **TODO** are yours; everything else is provided.\n",
    "\n",
    "**References.** Corbet (1984) A&A 141, 91 · Knigge, Coe & Podsiadlowski (2011) Nature 479, 372 · Silverman (1981) J. R. Stat. Soc. B 43, 97 ·\n",
    "Peterson (2001), *Variability of AGN*, §4 · White & Peterson (1994) PASP 106, 879 · Edelson & Krolik (1988) ApJ 333, 646 · Peterson et al. (1998) PASP 110, 660 ·\n",
    "Vincentelli et al. (2021) MNRAS 504, 4337."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a978a90b",
   "metadata": {},
   "source": [
    "## 0. Toolkit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a71a815",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np, csv, os\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy import stats\n",
    "import warnings; warnings.filterwarnings(\"ignore\", category=UserWarning)\n",
    "plt.rcParams.update({\"figure.figsize\": (9, 4), \"font.size\": 11})\n",
    "rng = np.random.default_rng(2026)\n",
    "DATA = \".\"          # folder that contains HMXB/, Mrk110_Swift/, Mrk110_ground/\n",
    "\n",
    "def load_periods(fn, cspin, corb):\n",
    "    # read a CSV with spin and orbital periods; empty fields -> NaN\n",
    "    P, O = [], []\n",
    "    for row in csv.DictReader(open(fn)):\n",
    "        P.append(float(row[cspin]) if row[cspin] else np.nan)\n",
    "        O.append(float(row[corb]) if row[corb] else np.nan)\n",
    "    return np.array(P), np.array(O)\n",
    "\n",
    "mwP, mwO = load_periods(f\"{DATA}/HMXB/MW_Ps_Porb.csv\", \"Spin_period\", \"Period\")\n",
    "smP, smO = load_periods(f\"{DATA}/HMXB/smc_Ps_Porb.csv\", \"period\", \"Porb\")\n",
    "print(\"Galaxy: rows\", mwP.size, \" with P_spin\", np.isfinite(mwP).sum(), \" with both\", (np.isfinite(mwP) & np.isfinite(mwO)).sum())\n",
    "print(\"SMC   : rows\", smP.size, \" with P_spin\", np.isfinite(smP).sum(), \" with both\", (np.isfinite(smP) & np.isfinite(smO)).sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c70ee09",
   "metadata": {},
   "source": [
    "---\n",
    "## A. The Corbet diagram\n",
    "\n",
    "Be/X-ray binaries are thought to sit near **spin equilibrium**, where the magnetospheric radius equals the corotation radius:\n",
    "$P_{\\rm eq}\\propto B^{6/7}\\dot M^{-3/7}$. A wider orbit means a lower mean $\\dot M$ and therefore a longer $P_{\\rm spin}$ — hence a\n",
    "correlation between $P_{\\rm spin}$ and $P_{\\rm orb}$ (Corbet 1984). Supergiant systems (wind-fed or Roche-lobe overflow) follow different physics.\n",
    "\n",
    "The files contain only the two periods. We use two crude cuts:\n",
    "* $P_{\\rm spin}<1$ s: rotation-powered pulsars in Be binaries (e.g. PSR B1259−63) — **not accreting, exclude**;\n",
    "* $P_{\\rm orb}\\le 10$ d: proxy for supergiant / Roche-lobe systems."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f5da441",
   "metadata": {},
   "source": [
    "**A1.** Make the Corbet diagram (log–log) for both galaxies. Mark the $P_{\\rm orb}\\le10$ d systems and the rotation-powered pulsars with different symbols."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ab2eeda",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: boolean masks for rows with both periods, and for P_spin < 1 s in the Galaxy\n",
    "okmw = ...\n",
    "oksm = ...\n",
    "rot  = ...\n",
    "# TODO: scatter P_spin vs P_orb on log-log axes, different symbols for P_orb <= 10 d and for rot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f18d03e",
   "metadata": {},
   "source": [
    "**A2.** Compute Pearson $r$ (on $\\log P$), Spearman $\\rho_s$ and Kendall $\\tau$ with their p-values for four samples:\n",
    "SMC all, SMC $P_{\\rm orb}>10$ d, Galaxy all ($P_{\\rm spin}>1$ s), Galaxy $P_{\\rm orb}>10$ d.\n",
    "Which coefficient would you quote, and why do the two galaxies differ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56a49e6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "samples = {\n",
    "    \"SMC all\":          (smP[oksm], smO[oksm]),\n",
    "    # TODO: add the other three samples\n",
    "}\n",
    "for k, (P, O) in samples.items():\n",
    "    x, y = np.log10(O), np.log10(P)\n",
    "    # TODO: stats.pearsonr, stats.spearmanr, stats.kendalltau -> print coefficient and p-value"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47efee17",
   "metadata": {},
   "source": [
    "**A3.** Permutation test. For the SMC, shuffle $P_{\\rm spin}$ among the systems 10 000 times and compute Kendall's $\\tau$ each time.\n",
    "What fraction of shuffles gives $|\\tau|\\ge|\\tau_{\\rm obs}|$? Compare with the p-value of A2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1606c35",
   "metadata": {},
   "outputs": [],
   "source": [
    "P, O = samples[\"SMC all\"]; x, y = np.log10(O), np.log10(P)\n",
    "tobs = stats.kendalltau(x, y)[0]\n",
    "# TODO: tperm = array of tau for 10000 permutations of y; p = fraction with |tau| >= |tobs|"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c49f26c9",
   "metadata": {},
   "source": [
    "**A4.** Which line? For SMC $P_{\\rm orb}>10$ d fit $\\log P_{\\rm spin}$ vs $\\log P_{\\rm orb}$ with OLS($y|x$), OLS($x|y$) and the OLS bisector\n",
    "(Isobe et al. 1990, ApJ 364, 104). Bootstrap the pairs (2000×) to get the scatter of each slope. Compare with the prediction of a simple spin-equilibrium picture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "408c0140",
   "metadata": {},
   "outputs": [],
   "source": [
    "P, O = samples[\"SMC P_orb>10 d\"]; x, y = np.log10(O), np.log10(P)\n",
    "def slopes(x, y):\n",
    "    # TODO: return OLS(y|x), OLS(x|y) and bisector slopes\n",
    "    ...\n",
    "# TODO: bootstrap pairs 2000 times, print slope ± std for each"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0b47a8d",
   "metadata": {},
   "source": [
    "**A5.** Do SMC and Galactic pulsars have the same $P_{\\rm spin}$ distribution? Use $\\log P_{\\rm spin}$ of all accreting pulsars ($P_{\\rm spin}>1$ s for the Galaxy).\n",
    "Plot the two empirical CDFs, mark the KS distance $D$, and compute the p-values of KS, Cramér–von Mises, Anderson–Darling and Mann–Whitney U.\n",
    "Calibrate the KS p-value with a permutation test (5000 shuffles of the pooled sample)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6dc9d701",
   "metadata": {},
   "outputs": [],
   "source": [
    "A = np.log10(smP[np.isfinite(smP)]); B = np.log10(mwP[np.isfinite(mwP) & (mwP > 1)])\n",
    "# TODO: stats.ks_2samp, stats.cramervonmises_2samp, stats.anderson_ksamp, stats.mannwhitneyu\n",
    "# TODO: permutation p-value of the KS statistic\n",
    "# TODO: plot both ECDFs and mark D"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc4e1e9f",
   "metadata": {},
   "source": [
    "**A6.** Is the SMC $P_{\\rm spin}$ distribution bimodal? Replace every pulsar by a Gaussian of width $h$ (in dex) — a **kernel density estimate**\n",
    "$$\\hat f_h(x)=\\frac{1}{Nh}\\sum_i\\phi\\!\\left(\\frac{x-x_i}{h}\\right).$$\n",
    "(a) Plot $\\hat f_h$ for $h=0.08,\\,0.2,\\,0.32,\\,0.6$ dex. (b) Choose $h$ by maximising the leave-one-out log-likelihood and compare with Silverman's rule\n",
    "$h=0.9\\min(s,\\,{\\rm IQR}/1.34)\\,N^{-1/5}$. (c) Count the modes as a function of $h$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8c18514",
   "metadata": {},
   "outputs": [],
   "source": [
    "def kde(v, g, h):\n",
    "    # TODO: sum of Gaussians of width h centred on the points v, evaluated on grid g, normalised\n",
    "    ...\n",
    "def nmodes(v, h, g=np.linspace(-1, 5.5, 1300)):\n",
    "    # TODO: number of local maxima of kde(v, g, h)\n",
    "    ...\n",
    "def loo(v, h):\n",
    "    # TODO: leave-one-out log-likelihood: sum_i log fhat_{-i}(x_i)\n",
    "    ...\n",
    "# TODO: (a) plot the KDE for several h; (b) h_CV and Silverman's h; (c) modes vs h"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c2aa8ed",
   "metadata": {},
   "source": [
    "**A7.** Silverman's (1981) mode test. Let $h_{\\rm crit}$ be the smallest $h$ for which $\\hat f_h$ has one mode.\n",
    "Draw 500 samples from the one-mode KDE at $h_{\\rm crit}$ (smoothed bootstrap: resample, add $h_{\\rm crit}\\,\\varepsilon$, rescale to the sample variance)\n",
    "and count how often they show more than one mode at $h_{\\rm crit}$. That fraction is the p-value of \"unimodal\". Do the same for the Galaxy.\n",
    "Compare with a 1- vs 2-Gaussian mixture (`sklearn.mixture.GaussianMixture`, ΔBIC)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "612fd367",
   "metadata": {},
   "outputs": [],
   "source": [
    "def hcrit(v, k=1):\n",
    "    # TODO: bisection in h for the smallest h with <= k modes\n",
    "    ...\n",
    "def silverman_test(v, k=1, nb=500):\n",
    "    # TODO: smoothed bootstrap from the KDE at h_crit; return h_crit and the fraction with > k modes\n",
    "    ...\n",
    "# TODO: run for SMC and Galaxy; compare with GaussianMixture(1) vs (2) BIC"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b75ce3b0",
   "metadata": {},
   "source": [
    "**A8 (discussion).** Write three sentences: is there a Corbet relation in each galaxy? Is the SMC spin distribution bimodal?\n",
    "What systematics (selection, classification, mixed populations) would you need to control before publishing either statement?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b98b67fd",
   "metadata": {},
   "source": [
    "---\n",
    "## B. Continuum lags in Mrk 110\n",
    "\n",
    "Swift monitored the narrow-line Seyfert 1 Mrk 110 about three times per day from 26 Oct 2017 to 25 Jan 2018 (Vincentelli et al. 2021);\n",
    "ground-based telescopes observed the optical bands. If the X-ray corona illuminates the accretion disc, the UV/optical should follow the X-rays,\n",
    "with a lag that grows with wavelength (thin disc: $\\tau\\propto\\lambda^{4/3}$).\n",
    "\n",
    "Files: `MJD  flux  error` (UVOT: erg cm⁻² s⁻¹ Å⁻¹; XRT: count s⁻¹). Ground files of 2017 give **MJD − 50000**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "229f5d1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "SW = {\"X\": \"XRT\", \"W2\": \"UVW2 1928\", \"M2\": \"UVM2 2246\", \"W1\": \"UVW1 2600\", \"U\": \"U 3465\", \"B\": \"B 4392\", \"V\": \"V 5468\"}\n",
    "WL = {\"W2\": 1928, \"M2\": 2246, \"W1\": 2600, \"U\": 3465, \"B\": 4392, \"V\": 5468, \"g\": 4770, \"Vg\": 5450, \"r\": 6215, \"i\": 7545, \"z\": 8700}\n",
    "def load_swift(b, t0=58050, t1=58150):\n",
    "    d = np.loadtxt(f\"{DATA}/Mrk110_Swift/Mrk110_{b}.dat\"); d = d[np.argsort(d[:, 0])]\n",
    "    return d[(d[:, 0] > t0) & (d[:, 0] < t1)]\n",
    "def load_ground(b, t0=58040, t1=58160):\n",
    "    d = np.loadtxt(f\"{DATA}/Mrk110_ground/Mrk110_{'V' if b == 'Vg' else b}_ground_2017.dat\"); d[:, 0] += 50000\n",
    "    d = d[np.argsort(d[:, 0])]; return d[(d[:, 0] > t0) & (d[:, 0] < t1)]\n",
    "LC = {b: load_swift(b) for b in SW}\n",
    "LC.update({b: load_ground(b) for b in [\"g\", \"Vg\", \"r\", \"i\", \"z\"]})\n",
    "for b, d in LC.items(): print(f\"{b:3s} N = {len(d):3d}   MJD {d[0,0]:.1f} – {d[-1,0]:.1f}   median Δt = {np.median(np.diff(d[:,0])):.2f} d\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26676c72",
   "metadata": {},
   "source": [
    "**B1.** Plot all Swift light curves (flux / mean) on a common time axis. Compute the fractional variability\n",
    "$F_{\\rm var}=\\sqrt{S^2-\\langle\\sigma^2\\rangle}\\,/\\,\\langle F\\rangle$ of each band. Why does it fall with wavelength?\n",
    "Look at UVW2 and UVM2 around MJD 58088–58106: what are the isolated low points?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f63d15fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: stacked plot of flux/mean for every Swift band\n",
    "def fvar(f, e):\n",
    "    # TODO: fractional variability with the noise variance removed\n",
    "    ..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6335f30",
   "metadata": {},
   "source": [
    "**B2.** Write the **ICCF** (Gaskell & Peterson 1987; White & Peterson 1994): for each lag $\\tau$, pair every observed $x(t_i)$ with the linearly\n",
    "interpolated $\\tilde y(t_i+\\tau)$ (only where $t_i+\\tau$ lies inside the $y$ coverage), compute Pearson's $r$ with the means and σ of the contributing points;\n",
    "repeat interpolating $x$ at $t_j-\\tau$ and average the two. Apply it to UVW2 → V and XRT → UVW2 for $-6\\le\\tau\\le6$ d.\n",
    "Sign convention: $\\tau>0$ means the second series lags the first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c957338e",
   "metadata": {},
   "outputs": [],
   "source": [
    "LAGS = np.arange(-6, 6.001, 0.1)\n",
    "def iccf(ta, a, tb, b, lags=LAGS, nmin=6):\n",
    "    out = np.full(lags.size, np.nan)\n",
    "    for k, L in enumerate(lags):\n",
    "        # TODO: mask of ta+L inside [tb[0], tb[-1]]; r1 = corr(a, interp(b at ta+L))\n",
    "        # TODO: mask of tb-L inside [ta[0], ta[-1]]; r2 = corr(interp(a at tb-L), b)\n",
    "        # TODO: out[k] = mean of r1, r2\n",
    "        ...\n",
    "    return out\n",
    "# TODO: ICCF of UVW2 -> V and XRT -> UVW2; plot r(τ)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1e52b67",
   "metadata": {},
   "source": [
    "**B3.** Write the **DCF** (Edelson & Krolik 1988): for all pairs $(i,j)$, $\\mathrm{UDCF}_{ij}=(x_i-\\bar x)(y_j-\\bar y)/\\sqrt{(s_x^2-\\langle e_x^2\\rangle)(s_y^2-\\langle e_y^2\\rangle)}$;\n",
    "average in bins of $\\Delta t_{ij}=t_j-t_i$ of width $\\Delta\\tau$, with error ${\\rm std}/\\sqrt{M-1}$. Overplot it on the ICCF of XRT → UVW2 for $\\Delta\\tau=0.5$ and 1 d.\n",
    "How many pairs fall in each bin?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a876db4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def dcf(ta, a, ea, tb, b, eb, lags, bw):\n",
    "    # TODO: standardised fluxes (noise variance removed), all pair products and time differences\n",
    "    # TODO: for each lag bin: mean, std/sqrt(M-1), and number of pairs M\n",
    "    ...\n",
    "# TODO: overplot the DCF (two bin widths) on the XRT -> UVW2 ICCF"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4efe4eea",
   "metadata": {},
   "source": [
    "**B4.** Peak and centroid. Find $\\tau_{\\rm peak}$ and $\\tau_{\\rm cent}$ (centroid of the contiguous region around the peak with $r\\ge0.8\\,r_{\\rm max}$) for UVW2 → V.\n",
    "What fraction of the lag window has $r\\ge0.8\\,r_{\\rm max}$? What does that do to the centroid?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dde056c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "def centroid(lags, r, frac=0.8):\n",
    "    # TODO: return (tau_cent, tau_peak, r_max)\n",
    "    ...\n",
    "# TODO: apply to r_WV; print the fraction of lags with r >= 0.8 r_max"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1952dde7",
   "metadata": {},
   "source": [
    "**B5.** Detrending. Subtract from every light curve its 10-d running mean (boxcar, $\\pm5$ d) and repeat B2/B4 for UVW2 → V, UVW2 → UVW1 and XRT → UVW2.\n",
    "What does the filter keep and what does it remove? Why does the CCF become narrower (remember CCF $=\\Psi*$ACF)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "021adead",
   "metadata": {},
   "outputs": [],
   "source": [
    "def boxcar(t, f, w=10.0):\n",
    "    # TODO: running mean within ±w/2 of each point\n",
    "    ...\n",
    "# TODO: D = detrended copies of all light curves; compare raw and detrended ICCFs for three pairs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b98814e",
   "metadata": {},
   "source": [
    "**B6.** FR/RSS errors (Peterson et al. 1998). For each realisation: draw $N$ epochs with replacement and keep each epoch once (≈63% survive),\n",
    "add Gaussian noise with the quoted errors, recompute the ICCF and its peak and centroid. Use 200 realisations for UVW2 → V (raw: report the **peak**)\n",
    "and for detrended UVW2 → UVW1 (report the **centroid**). Quote median and 68% range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ce45260",
   "metadata": {},
   "outputs": [],
   "source": [
    "def frrss(A_, B_, n=200):\n",
    "    # TODO: random subset (unique indices) + flux randomisation, ICCF, store centroid and peak\n",
    "    # TODO: return 16/50/84 percentiles of the centroid and of the peak\n",
    "    ...\n",
    "# TODO: UVW2 -> V (raw, peak) and UVW2 -> UVW1 (detrended, centroid)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "653e3621",
   "metadata": {},
   "source": [
    "**B7.** The lag spectrum. Measure the raw $\\tau_{\\rm peak}$ (with FR/RSS) of every band against UVW2 — Swift UVM2…V **and** ground g, V, r, i, z.\n",
    "Plot lag vs wavelength and fit $\\tau=A\\left[(\\lambda/1928\\,\\text{Å})^{4/3}-1\\right]$ by weighted least squares. Where do lags appear?\n",
    "Compare with the ≈2 d at 9100 Å reported by Vincentelli et al. (2021). *(≈2–3 min.)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47a6fcc2",
   "metadata": {},
   "outputs": [],
   "source": [
    "res = {}\n",
    "# TODO: for every band b: res[b] = FR/RSS percentiles of the raw peak lag of UVW2 -> b\n",
    "# TODO: weighted fit of tau = A[(lam/1928)^(4/3) - 1]; plot lag vs wavelength with the fit"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82ffb224",
   "metadata": {},
   "source": [
    "**B8.** Is the X-ray–UV correlation significant? Simulate 200 pairs of **independent** red-noise light curves (damped random walk, $\\tau_{\\rm DRW}=20$ d)\n",
    "sampled exactly like XRT and UVW2, detrend them as in B5, and record the maximum ICCF for each pair. Where does the observed detrended $r_{\\rm max}$ of XRT → UVW2 fall?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "973b6550",
   "metadata": {},
   "outputs": [],
   "source": [
    "def drw(t, tau=20.0, sig=1.0):\n",
    "    # TODO: damped random walk (Ornstein-Uhlenbeck) at times t\n",
    "    ...\n",
    "# TODO: 200 independent pairs with the XRT and UVW2 sampling, detrended; max ICCF of each; compare with the data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fc8d5e6",
   "metadata": {},
   "source": [
    "**B9 (discussion).** In five sentences: what lags did you measure, in which bands, with which method? What did detrending change?\n",
    "Is the UV driven by the X-rays in this campaign? What would you need (cadence, bands, length) to do better?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
